Table Of Contents:

Last Updated September 30th , 2011

Main Focus

Testing Framework and Testing

  • Jens and Gaurang will work on setting up a Bamboo Framework

FHS Work

  • Main JIRA item
  • Moving pegasus to native Debian and Redhat packaging
  • Change of all tools to pick up new layout.
  • kickstart/seqexec/keg to be renamed to have a pegasus prefix
  • Planner code needs to be changed to reflect new naming and layout.

Pegasus Lite and Improving Condor IO Staging Mode

  • Main JIRA item
  • Notion of Staging Site in the Planner
  • Staging of worker packages for non shared filesystem
  • Pegasus-Lite- Shell launcher for Jobs on remote sites
  • New clients for create dir and cleanup jobs that can run locally?
    • for create dir , transfer of zero byte file
  • Do away with the S3 specific implementations
  • Do away with Condor Refiners for Condor IO and fold it to the SLS interfaces.

Monitord

  • refactoring of monitord code. right now it is all in one file!
  • Locking mechanism to ensure that multiple instances of monitord cannot run on same submit directory
  • boolean option whether to store stdout or stderr in the database

Stampede Related Activities

  • Main JIRA item
  • Porting of pegasus analyzer
  • Reading of DAGMan generated jobstate.log files
  • pegasus-statistics changes
    • Option to expose stats as a csv file?
    • pegasus-statistics should take in multiple workflow id's / pass the list of directories
    • statistics . size of task edge tables. can the size be decreased. we need to talk to stampede folks more about it.
  • New Plots and Statistics?
  • CPU Time tracking in addition to the walltime ( as currently reported by kickstart ).
      • Jens says utime is in the kickstart record
      • should we change the schema ?
  • Upgrading the stampede schema's
    • how do we handle the upgrade to handle extra things
  • multicore jobs calculations.
    • there should be a multiplier /multiplying factor that is propagated through the system.
    • pegasus can put the number of cores in the submit file
  • storing status vs exitcode
    • if a job dies on a signal , can we have a special exitcode so that kickstart record has an exitcode
    • in the stampede schema , monitord should only populating raw status in the db
    • pegasus analyzer should be smarter about whether failure happened because of signal and or a job failed by exitcode.
    • put in an extra column in the stampede schema ???

Other Pegasus Changes

  • Hierarichal Workflows
    • Propagation of options from command line to the sub workflows.
  • Time based clustering
    • Similar to horizontal clustering.
    • runtime profile key that can be used with the jobs.
    • another profile key that says the clusters.maxruntime.

Other Things

  • Option to do a mv on stageout . Brain workflows usecase.
    • to do in the transfer client not necessary in the planner. needs more thought and discussion

Porting the VM to 3.2.0

People Involved [Karan,Rajiv]

  • Addition of new exercises
    • How to add notifications
  • No labels