Last Updated March 25th April 5th , 2011

Main Focus

• Add notification support to Pegasus
• Moving auxillary tools to stampede db
• Other Stampede Related Changes
• Usability Changes
• User Guide Reorganization 
• Testing Framework and Testing.

JIRA Task Board

TaskBoard for 3.1

Items for April End

Stampede Related Changes

• Main JIRA Issue
  • Main JIRA Item PM-345
• Updated Database Schema
  • Schema Page
• Additional DB schema changes to be able to connect jobs/tasks in the DAX with corresponding kickstart records [Karan,Fabio,Monte]
• Identify the new events for the updated schema [Karan,Fabio]
  • Pegasus to generate netlogger events to a file in the submit directory [Karan]
  • Changes to monitord to conform to new schema, populate netlogger stream from Pegasus. [Fabio]
• Addition of workflow metrics file containing distribution of jobs into the DB [Fabio,Karan]
  • Revive the metrics file created by Pegasus . Should be populated in the submit directory. [Karan]

Notification Support in Pegasus via monitord

• unmigrated-wiki-markupMain JIRA Issue
  • Main JIRA Item PM-346
• Monitord needs to be managed by Condor. What happens if monitord crashes or condor/system crashes? We want monitord to come up automatically as Condor recovers after a restart. *\[Fabio,Gaurang\]*\*
• Wiki MarkupMonitord needs to support notifications *\[Fabio\]*
• Wiki MarkupRequires changes to Pegasus to generate input file for monitord *\[Rajiv,Karan\]*
• Wiki MarkupCome up with default notify scripts in the toolkit that notify the user and generate some status reports. *\[Gaurang\]*
• Changes to DAX Schema
  • Main JIRA Item PM-350
  • Addition of invoke element at the workflow level
  • Wiki MarkupChanges to python API *\[Gideon\]*unmigrated-wiki-markup
  • Changes to Perl API *\[Jens\]*unmigrated-wiki-markup
  • Changes to JAVA API *\[Gaurang\]* Wiki Markup
  • Change to JAVA Parser *\[Karan\]*

  • Wiki Markup
    Instead of pegasus-run launching monitord, monitord should appear as an independent job in the workflow with the highest priority *\[Karan\]*

  • Wiki MarkupFabio needs to make sure exitcodes are thrown correctly and restarts are handled correctly. *\[Fabio\]*

...

Monitord

...

Management

...

[Fabio,Gaurang

...

]

...

https://confluence.pegasus.isi.edu/display/pegasus/Monitord+Management+via+Condor

Auxillary Tools to Stampede DB

• Wiki Markuppegasus-statistics *\[Prasanth\]*unmigrated-wiki-markup, Mats]
• pegasus-plots *\[Prasanth\]* Wiki Markup
• pegasus-analyzer *\[Fabio\]*

...

Monitord

...

Changes

...

[Fabio

...

]

...

• Monitord also needs to be able to account for newer versions of Condor DAGMan creating a jobstate.log file.

Stampede Related Changes

...

Additional DB schema changes to be able to connect jobs/tasks in the DAX with corresponding kickstart records *\[Fabio,Monte\]*

Addition of workflow metrics file containing distribution of jobs into the DB *\[Fabio,Karan\]*

...

Revive the metrics file created by Pegasus . Should be populated in the submit directory. *\[Karan\]*

...

DAX API changes

• Feedback from Duncan brown while using Python API
• Executable handling in Python API and JAVA API
  • internally executables handled as lists not as Sets
  • python api does not allow for adding edges based on id's.
  • also have a getJob function based on ID.
  • escape function in python api?is it for all strings

• JAVA API [Gaurang]
• Python API [Gideon]
• Perl API [Jens]

S3 support [Mats, Karan]

• pegasus-transfer to support pegasus-s3 tool Gideon wrote [Mats]
• changes to Pegasus to use pegasus-transfer for S3 [Karan]
  • handle transfer of S3 config file etc
  • get s3 to work with SeqExec launcher.

Usability Changes

• refactoring of clustered jobs (internal to pegasus)
• Addition of -conf option
  unmigrated-wiki-markup
  • Java Clients *\[Prasanth,Rajiv,Karan\]* Wiki Markup
  • Python Clients *\[Prasanth,Fabio\]*
    • pegasus-statistics, pegasus-plots, pegasus-analyzer, monitord
    unmigrated-wiki-markup
  • Perl Clients *\[Gaurang\]*
  Wiki Markup
• Improvements to pegasus-tc-client *\[Prasanth\]*
  • the pegasus-tc-client output is in the old deprecated format.
  Wiki Markup
• Improvements to pegasus-rc-client *\[Rajiv\]*
  • Investigation of RLS compatibility issues
• Addition of default categories to allow for easier specification of category based knobs at DAGMan level
  • cleanup jobs
  • subdax jobs
• Wiki MarkupImprove Rescue DAG semantics *\[Rajiv\]* [Rajiv]
• Pegasus should not require the jobmanager compute to be present in site catalog. [Karan] https://jira.isi.edu/browse/PM-277

Condor Common Log Handling?

• Condor Common Log Handling to be discussed. https://jira.isi.edu/browse/PM-222

Open Question

Improve the Condor File IO mode in Pegasus ? Not clear how to do it without going down the staging-sites option.
User experience can be improved, but would be a hack to do it in Pegasus without staging-sites option.

...

User

...

Guide

...

Reorganization

...

[Bill

...

]

• Dependant on Bill

Testing Framework and Testing

...

*People Involved \ [Jens,Gaurang\]*

Porting the VM to 3.1.0

Wiki Markup*People Involved \ [Karan,Rajiv\]*

• Addition of new exercises

...

• Fabio gets stampede related changes done end of april.
• Fabio gets whole of may to put in notification support.
• End of May we may have a first beta, where everything idenitifed above is done.
• June is spent testing on the release .