To dos
- pegasus-run (gaurang) DONE
- pegasus-plan (gaurang) DONE
- DAX format (gaurang) DONE
- profiles (karan) DONE
- properties (karan) DONE
- cleanup of rls client (karan) DONE
- documentation URGENT TODO
- overall (Ewa)
- quickstart, admins (Gaurang)
- different grids, pegasus internal guides (Karan)
- redo the code structure, rename the name space org.griphyn/cplanner to isi.edu/pegasus MOVED TO 2.1 release
Pegasus Tools
pgs for profiles namespaces and properties
pegasus-plan
- remove selection for pegasus and euryale
- default vo group should be pegasus
- simplifying the options are parsed
- keep pegasus-plan and gencdag in sync
pegasus-run
This will plan and submit the dag
- rename
pegasus-submit-dag
- old vds-run equivalent
pegasus-delete
- run the cleanup dag even on a workflow that is running (workflow can be partially executed)
will run condor-rm and run the cleanup dag
pegasus-status
- wrapper around dagman.out file
pegasus-partition
- rename partition-dax
pegasus-version
pegasus-config
- check if can reuse vds-config
.pgsrc
- tailstad needs to pick up from .pgsrc
$PEGASUS
graphing tools will be better documented, moved to the primary bin directory, pie tools
XML format for TC--ask Jens for his format
- for now loose the shell wrapper: genmdag
CVS cleanup
- remove euryale
- keep vdl but put it in contrib
Documentation cleanup
- move to docbook
- rename profiles
- revisit properties names
Order of cleanup
- Quick start guide – Gaurang
- Running on different Grids
- Condor
- Condor-G
- GT4
- Glide-ins
- FAQ
- Administrators Guide
- Workflow Clustering
- Transfer Configurations
- Introduction to Pegasus an Architecture Guide--- Ewa
- Replica Selection (maybe have something for advanced features)
- Staging of Executable
- Site Selection
- Getting started with the Grid
- VDL--may not require changes
- LIGO should have its own help sheet (for example for the info on file naming)
Set up Bugzilla for Pegasus
Changes to Pegasus
- remove old rls local service stuff
- use the new client for using rls
- change cache file format
- change DAX format to use Pegasus namespace
- change code to pick Pegasus worker binaries automatically from $PEGASUS set in the site catalog (2.0 or 2.1 )
Accounting features
Ensure wiki backup
done
Purely local-non-db installation of Pegasus
- make tailstatd work with embedded db
allow users to include physical file names in DAX
Integrated workflow db
Workflow Provisioning
- Work on an easier way of integrating glide-in to provision resources.
Currently glide in's are hard to setup. Make it easier to use out of the box. - Is it possible to integrate Gurmeet's research here?
- Also an interesting thing to look at is using provisioning to drive the partitioning process.
- A look at glidein factory. [ Glidein Factory|http://home.fnal.gov/~sfiligoi/glideinWMS/doc/manual/factory/index.html]
- Some OSG related notes on glidein are here
Apply for cycles on the TG
Functionality Improvements
- Add data cleanup capabilities . Done
- Support for versioning in the transformation catalog.
- Deal with data missing but not in RLS
- Provide an option to register input data products
- Monitoring tools
- Dealing with issues of managing multiple transitions
To Do After the Release
- Add stage-out clustering
- Make sure that pegasus is run on full hipe without partitioning.
- Improve the cleanup algorithm. Make a separate cleanup job only when absolutely necessary
- Fix cleanup over deferred planning
- Create a script to use rescue dags before replanning while in deferred planning mode.
New Ventures
Ensemble Manager
Pegasus in the browser with workflow construction and everything else done within the browser, would let people do workflows without downloading any software.
Metronome on the TG--dynamic build and test environment, maybe we can use it for building application codes for various platforms, in order to disperse the code into the environment.
Papers
SciProg paper with LIGO
Book chapter early April