To dos

  1. pegasus-run (gaurang) DONE
  2. pegasus-plan (gaurang) DONE
  3. DAX format (gaurang) DONE
  4. profiles (karan) DONE
  5. properties (karan) DONE
  6. cleanup of rls client (karan) DONE
  7. documentation URGENT TODO
    1. overall (Ewa)
    2. quickstart, admins (Gaurang)
    3. different grids, pegasus internal guides (Karan)
  8. redo the code structure, rename the name space org.griphyn/cplanner to MOVED TO 2.1 release

Pegasus Tools

pgs for profiles namespaces and properties


  • remove selection for pegasus and euryale
  • default vo group should be pegasus
  • simplifying the options are parsed
  • keep pegasus-plan and gencdag in sync


This will plan and submit the dag

  • rename


  • old vds-run equivalent


  • run the cleanup dag even on a workflow that is running (workflow can be partially executed)
    will run condor-rm and run the cleanup dag


  • wrapper around dagman.out file


  • rename partition-dax



  • check if can reuse vds-config


  • tailstad needs to pick up from .pgsrc


graphing tools will be better documented, moved to the primary bin directory, pie tools

XML format for TC--ask Jens for his format

  • for now loose the shell wrapper: genmdag

CVS cleanup

  • remove euryale
  • keep vdl but put it in contrib

Documentation cleanup

  • move to docbook
  • rename profiles
  • revisit properties names

Order of cleanup

  1. Quick start guide – Gaurang
  2. Running on different Grids
    1. Condor
    2. Condor-G
    3. GT4
    4. Glide-ins
  3. FAQ
  4. Administrators Guide
  5. Workflow Clustering
  6. Transfer Configurations
  7. Introduction to Pegasus an Architecture Guide--- Ewa
  8. Replica Selection (maybe have something for advanced features)
  9. Staging of Executable
  10. Site Selection
  11. Getting started with the Grid
  12. VDL--may not require changes
  1. LIGO should have its own help sheet (for example for the info on file naming)

Set up Bugzilla for Pegasus

Changes to Pegasus

  • remove old rls local service stuff
  • use the new client for using rls
  • change cache file format
  • change DAX format to use Pegasus namespace
  • change code to pick Pegasus worker binaries automatically from $PEGASUS set in the site catalog (2.0 or 2.1 )

Accounting features

Ensure wiki backup


Purely local-non-db installation of Pegasus

  • make tailstatd work with embedded db
    allow users to include physical file names in DAX

Integrated workflow db

Workflow Provisioning

  • Work on an easier way of integrating glide-in to provision resources.
    Currently glide in's are hard to setup. Make it easier to use out of the box.
  • Is it possible to integrate Gurmeet's research here?
  • Also an interesting thing to look at is using provisioning to drive the partitioning process.
  • A look at glidein factory. [ Glidein Factory|]
  • Some OSG related notes on glidein are here

Apply for cycles on the TG

Functionality Improvements

  • Add data cleanup capabilities . Done
  • Support for versioning in the transformation catalog.
  • Deal with data missing but not in RLS
  • Provide an option to register input data products
  • Monitoring tools
  • Dealing with issues of managing multiple transitions

To Do After the Release

  • Add stage-out clustering
  • Make sure that pegasus is run on full hipe without partitioning.
  • Improve the cleanup algorithm. Make a separate cleanup job only when absolutely necessary
  • Fix cleanup over deferred planning
  • Create a script to use rescue dags before replanning while in deferred planning mode.

New Ventures

Ensemble Manager

Pegasus in the browser with workflow construction and everything else done within the browser, would let people do workflows without downloading any software.

Metronome on the TG--dynamic build and test environment, maybe we can use it for building application codes for various platforms, in order to disperse the code into the environment.


SciProg paper with LIGO

Book chapter early April

  • No labels