Developer Meetings

June 2013

June 17th, 2013

• Pegasus Development
  • the dax job handling is completed.
  • update on ligo front.
  • condor priorities for local universe jobs
    • not handled right now.
    • gideon has a ticket open for them.
  • gideon observation of s3
    • scalable but not good latency or
• Pegasus Lite Paper
  • mats is almost done with the runs. to grep through the runs to get the intermediate files in and out of S3
  • not done the S3 caching for rosetta as yet. still not sure. too much work for the time remaining.
  • mats did do the runs with task clustering. he got better numbers and saw a difference in case of rosetta.
  • interleaving of compute jobs and transfers. may help montage.. but won't help rosetta
  • whether we should include the new pegasus 4.2 features.
• Cleanup Algorithm
• Glacier Backups for NFS?
  • instead of using two qnaps, just have one and use other for duplicates
  • we need a place for backups
  • currently the QNAPS are 18TB each with raid 6. Raid 10 is a better configuration on the QNAP according to the forums. This means though we will have half the space.
    • have one qnap for scratch
    • have other qnap for storage - the storage will be backed upto glacier. right now QNAP only support S3. Support for glacier is coming.
  • ewa and richard think glacier backups are a good option.
    • there might be a purge policy required on glacier.
• Precip Paper
  • change tracking on
  • use dropbox
  • broadcast when you making a new version.

June 10th, 2013

- Pegasus Development

- change to dax handling

- fix of stdout 

- regex based replica catalog. 

- changes to pegasus-statistics for aggregate statistics

Pegasus  Lite Paper

- compute data between s3 and local disk.

- compute costs for the runs ? 

- have data outside 

- local cache for the S3 client ??  could affect the rosette cache. 

 - change the rosetta workflow.

 - if there are a lot of small files.

 - reading parts of files.

- Ewa will send her version of the changes.

Sudharshan Algorithm for Cleanup

• Greedy appraoch planned
• will try implementing a version and show the different executable workflows created

June 3rd, 2013

Pegasus Lite Paper

• Breakdown of the runtimes , experiments
  
  • In case of sharedfs, the kickstart runtimes in the breakdown file will be longer
  • for the S3 case we can calculate the S3 transfer time by calculating the difference between the cumulative runtimes
  • doing two experiments rosetta(cpu intensive) and montage( io intensive)

Pegasus Development

• Java DAX API issues
  
  • might be some bugs in there.

Precip Paper

• Ewa wants a link to pegasus website in the paper.
• have more logical thinking in the paper, like reliability and repeatability
• Sepideh adding some new figures to the paper.
• Maciek will provide an experiment use-case for the paper.

Stampede and Corral Annual Reports

• Karan and Mats will be working on these

Sudarshan's Project

• Going to look into providing a cleanup algorithm that meets a given storage constraint
• Will look at the static problem of inserting dependencies into the workflow to achieve a solution

PMC Paper

• on amazon
• with clustering and pmc

Shirts

• Should get the logo sample this week, once we approve then we can order shirts

dV/dT

• Rafael is working on a draft of the data collection and modeling paper
• We are planning on publishing data, will start drafting a format this week

May 2013

May 20th, 2013

Confluence is going slow. Mats is going to look.

Analytics are set up on Confluence now.

Pegasus Transfer

• Mats committed a new version that has support for 2-stage transfers

Pegasus S3 Client

• Gideon changed .s3cfg to .pegasus/s3cfg

Pegasus Lite Paper

• Mats is working on the experiments
• We have two weeks to the deadline

PMC Paper

• Experiments on Amazon comparing Pegasus, Pegasus w/ Clustering, PMC alone

Pegasus Service

• Finished setting up users and test suite
• Next is a quick-and-dirty ensemble manager implementation
• Gideon is going to commit a change to Pegasus that removes the dashboard components. They will live in the pegasus-service repository from now on.

Summer Student

• Need to think up a project. Needs to be research-oriented and relatively small.
• Cleanup? Precip? 

Contacting users

• Find out if they need anything.

Examples

• Simple examples in Perl, Python and Java
• Gideon will add them to the examples in the pegasus Git repo

April 2013

April 22nd, 2013

April 8th, 2013

April 1st, 2013

Pegasus Lite Paper

• Waiting on Ewa
• Not much we can do about the IEEE conference. The page limit is 8 , the current size of the paper.

XSEDE Poster

• Pegasus Poster. Karan will send update
• Also a joint Pegasus BOSCO poster
• Also as part of that we will get the MPI workflows up and running through Pegasus and BOSCO

Pegasus Development

• Bypass of staging input files for Pegasus Lite Case
• Inplace cleanup bug fixes done.
• pegasus-s3
  
  • gideon checked in changes of copy from one file to another
  • mats adds a pegasus transfer
• workflow cleanup nodes
  
  • separate cleanup node in the workflow
  • for hierarchal workflows we only delete the outermost workflow
  • what happens if no output-site specified
    • the ligo case!
• backward compatiblity for LIGO
• Pegasus Dashboard
  • general javascript updates
• Generic Pegasus Slides
  
  • 2-3 slides.

March 2013

March 25th, 2013

• Pegasus Lite Paper Submission
  
  • We will try for https://sites.google.com/site/sweetworkshop2013/
  • Karan will move the paper to the ACM format
• Pegasus-statisitcs
  
  • Waiting on Scott to get back with the list of metrics
  • Rajiv will be working on it
• pegasus-s3 changes
  
  • we want to be able to copy output files from one s3 bucket to another
  • requires changes to pegasus-transfer and pegasus-s3
• final node for cleaning up remote directories
  
  • also related is getting the cleanup algorithm working when we bypass first level staging.

March 18th, 2013

• Mats has an RPM almost sorted out for LIGO that does not require us to have PYTHONPATH set. Instead the libraries go into standard locations
• Karan is testing this RPM at on spice-dev1 and has setup a page with instructions on how to submit a test workflow to VIRGO
  
  • https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/aLIGOaVirgo/130307121559GeneralInstructions%20For%20Submitting%20Workflows%20on%20VIRGO
• Statistics across root workflows
  
  • earlier gaurang had generated statistics for scec runs by hand... executiing queries on the msql command line
  • he does not have the queries documented anywhere
  • this is something we have talked about in context of 4.3 with Rajiv
  • will follow up with scott on wednesday's call
• 4.2.1 release
  
  • backward compatibility for LIGO . still to be done
  • probably next week after the pegasus annual report
  • RPM to handle native python installation
• Pegasus Annual Report
  • Karan will work on it this week
  • Try to follow the same template as earlier.

March 4th, 2013

• Sent link on DAGMan metrics to DAGMan Metrics Reporting to Ewa
• Metrics for Rob Quick's workflow
• Gideon pushed out kickstart changes
• Rajiv has pushed changes to the queries for the dashboard.
• Setup meeting with Jaime and Derrick at OSG AHM to discuss
  • remote_initialdir
  • extra attributes for glite/bosco submissions
  • mpi workflows.
• OSG Poster to be made this week. And 4.2 Release slides.

February 2013

February 11th, 2013

Direct submission of workflows to PBS

• Glite submission in Condor. We setup a VM that hosts a PBS scheduler and using that too test
• Karan prepared an example for 4.2 that can be used to submit directly to local PBS using the glite interfaces in Condor
  • the remote_initialdir  / +remote_iwd  does not work
    
    • problem for MPI codes
    • for the time being, the example prepared relies on kickstart to change the directory before launching a job
  • there is also a ssh style that allows us to use BOSCO to do remote submissions using SSH to a PBS cluster
    • that one also has the issue of remote initialdir

 - jobstate.log refactoring. 

 - data transfer ( support for globus online) 

- lightweight tracing

 -  task stats. net link socket pegasus-kickstart . how much memory the task used and io used. 

 - add task stats to kickstart

 - ptrace

 - trace  linux equivalent is system tap

- dashboard improvements

 - single api for clients

 - last week drop down

 - performance run on large workflows.

February 4th, 2013

• CCGrid / Pegasus Lite Paper
  •  Performance section
  •  remove the experiments section?
  •  OR
  •  extra experiments section 
  •  have the squid proxy cache
  • find a workshop to submit the paper

• Cloud Paper
  
  •  Ewa is working on it.
    
    
• Git HUB Migration
  
  •  - couple of branches like monitord , pmc and dang are branches
  •  - svn will be made read only . 
  •  - update the website with all the development information
  •  - bamboo scripts
  •  - documentation ( long term )
  •  - nightly builds

• SSH Submission
  
  •  - gsissh submission for blue waters
  •  - ssh to blue waters is required for OTP
  •  - passing of parameters to PBS
  •  - SSH key
  •  - ssh agent.
  •  - queue keyword
  •  - Batch session
  •  - submit jobs to HPCC
  •  - Gideon will do that. 
    
    
• monitord memory explosion
  
  •  - long term for monitord 
  •  - pegasus-dagman replacement 
    
    
•   minor release 4.2.1
  
  •  - potential monitord bug issue
  •  - long term dagman replacement
    
    
• Response time for metrics page
  •  - occasionally it is slow