Page History

...

JIRA Task Board

TaskBoard for 3.1

Items for April End

Stampede Related Changes

• Main JIRA Issue
  • Main JIRA Item PM-345
• Updated Database Schema
  • Schema Page
• Wiki MarkupAdditional DB schema changes to be able to connect jobs/tasks in the DAX with corresponding kickstart records *\[Karan,Fabio,Monte\]*unmigrated-wiki-markup
• Identify the new events for the updated schema *\[Karan,Fabio\]*
  • Wiki MarkupPegasus to generate netlogger events to a file in the submit directory *\[Karan\]*unmigrated-wiki-markup
  • Changes to monitord to conform to new schema, populate netlogger stream from Pegasus. *\[Fabio\]*
  unmigrated-wiki-markup
• Addition of workflow metrics file containing distribution of jobs into the DB *\[Fabio,Karan\]*
  unmigrated-wiki-markup
  • Revive the metrics file created by Pegasus . Should be populated in the submit directory. *\[Karan\]*

Notification Support in Pegasus via monitord

• Main JIRA Issue
  • Main JIRA Item PM-346
• Wiki MarkupMonitord needs to be managed by Condor. What happens if monitord crashes or condor/system crashes? We want monitord to come up automatically as Condor recovers after a restart. *\[Fabio,Gaurang\]*
• Wiki MarkupMonitord needs to support notifications *\[Fabio\]*
• Wiki MarkupRequires changes to Pegasus to generate input file for monitord *\[Rajiv,Karan\]*
• Wiki MarkupCome up with default notify scripts in the toolkit that notify the user and generate some status reports. *\[Gaurang\]*
• Changes to DAX Schema
  • Main JIRA Item PM-350
  • Addition of invoke element at the workflow levelunmigrated-wiki-markup
  • Changes to python API *\[Gideon\]* Wiki Markup
  • Changes to Perl API *\[Jens\]* Wiki Markup
  • Changes to JAVA API *\[Gaurang\]* Wiki Markup
  • Change to JAVA Parser *\[Karan\]*
  • Wiki MarkupFabio needs to make sure exitcodes are thrown correctly and restarts are handled correctly. *\[Fabio\]*

...

Monitord

...

Management

...

[Fabio,Gaurang

...

]

...

https://confluence.pegasus.isi.edu/display/pegasus/Monitord+Management+via+Condor

Auxillary Tools to Stampede DB

...

• pegasus-statistics *\[Prasanth, Mats\]*
• Wiki Markuppegasus-plots *\[Prasanth\]*unmigrated-wiki-markup
• pegasus-analyzer *\[Fabio\]*

...

Monitord

...

Changes

...

[Fabio

...

]

...

• Monitord also needs to be able to account for newer versions of Condor DAGMan creating a jobstate.log file.

...

• Feedback from Duncan brown while using Python API
• Executable handling in Python API and JAVA API
  • internally executables handled as lists not as Sets
  • python api does not allow for adding edges based on id's.
  • also have a getJob function based on ID.
  • escape function in python api?is it for all strings
  unmigrated-wiki-markup

• JAVA API *\[Gaurang\]*unmigrated-wiki-markup
• Python API *\[Gideon\]* Wiki Markup
• Perl API *\[Jens\]*

...

S3

...

support

...

[Mats,

...

Karan

...

]

...

• Wiki Markuppegasus-transfer to support pegasus-s3 tool Gideon wrote *\[Mats\]*unmigrated-wiki-markup
• changes to Pegasus to use pegasus-transfer for S3 *\[Karan\]*
  • handle transfer of S3 config file etc
  • get s3 to work with SeqExec launcher.

...

• refactoring of clustered jobs (internal to pegasus)
• Addition of -conf option
  • Wiki MarkupJava Clients *\[Prasanth,Rajiv,Karan\]* Wiki Markup
  • Python Clients *\[Prasanth,Fabio\]*
    • pegasus-statistics, pegasus-plots, pegasus-analyzer, monitord
    unmigrated-wiki-markup
  • Perl Clients *\[Gaurang\]*
  Wiki Markup
• Improvements to pegasus-tc-client *\[Prasanth\]*
  • the pegasus-tc-client output is in the old deprecated format.
  unmigrated-wiki-markup
• Improvements to pegasus-rc-client *\[Rajiv\]*
  • Investigation of RLS compatibility issues
• Addition of default categories to allow for easier specification of category based knobs at DAGMan level
  • cleanup jobs
  • subdax jobs
• Wiki MarkupImprove Rescue DAG semantics *\[Rajiv\]*
• Wiki MarkupPegasus should not require the jobmanager compute to be present in site catalog. *\[Karan\]* [https://jira.isi.edu/browse/PM-277]

Condor Common Log Handling?

...

Improve the Condor File IO mode in Pegasus ? Not clear how to do it without going down the staging-sites option.
User experience can be improved, but would be a hack to do it in Pegasus without staging-sites option.

...

User

...

Guide

...

Reorganization

...

[Bill

...

]

• Dependant on Bill

Testing Framework and Testing

Wiki Markup*People Involved \ [Jens,Gaurang\]*

Porting the VM to 3.1.0

...

*People Involved \ [Karan,Rajiv\]*

• Addition of new exercises

...