- Add notification support to Pegasus
- Moving auxillary tools to stampede db
- Other Stampede Related Changes
- Usability Changes
- User Guide Reorganization
- Testing Framework and Testing.
JIRA Task Board
Stampede Related Changes
- Main JIRA Issue
- Updated Database Schema
Additional DB schema changes to be able to connect jobs/tasks in the DAX with corresponding kickstart records *\ Wiki Markup
- Additional DB schema changes to be able to connect jobs/tasks in the DAX with corresponding kickstart records [Karan,Fabio,Monte\]*unmigrated-wiki-markup
- Identify the new events for the updated schema *\[Karan,Fabio\]*
- Pegasus to generate netlogger events to a file in the submit directory *\[Karan\]*unmigrated-wiki-markup
- Changes to monitord to conform to new schema, populate netlogger stream from Pegasus. *\[Fabio\]*
- Addition of workflow metrics file containing distribution of jobs into the DB *\[Fabio,Karan\]*
- Revive the metrics file created by Pegasus . Should be populated in the submit directory. *\[Karan\]*
Notification Support in Pegasus via monitord
- Main JIRA Issue
Monitord needs to be managed by Condor. What happens if monitord crashes or condor/system crashes? We want monitord to come up automatically as Condor recovers after a restart. *\[Fabio,Gaurang\]*unmigrated-wiki-markup Wiki Markup
- Monitord needs to support notifications *\[Fabio\]*
Requires changes to Pegasus to generate input file for monitord *\[Rajiv,Karan\]*unmigrated-wiki-markup Wiki Markup
- Come up with default notify scripts in the toolkit that notify the user and generate some status reports. *\[Gaurang\]*
- Changes to DAX Schema
- Main JIRA Item PM-350
- Addition of invoke element at the workflow level
Changes to python API *\[Gideon\]* Wiki Markup Wiki Markup
- Changes to Perl API *\[Jens\]*unmigrated-wiki-markup
- Changes to JAVA API *\[Gaurang\]*unmigrated-wiki-markup
- Change to JAVA Parser *\[Karan\]*
Fabio needs to make sure exitcodes are thrown correctly and restarts are handled correctly. *\[Fabio\]* Wiki Markup
Auxillary Tools to Stampede DB
- pegasus-statistics *\[Prasanth, Mats\]*unmigrated-wiki-markup
- pegasus-plots *\[Prasanth\]*unmigrated-wiki-markup
- pegasus-analyzer *\[Fabio\]*
- Monitord also needs to be able to account for newer versions of Condor DAGMan creating a jobstate.log file.
- Feedback from Duncan brown while using Python API
- Executable handling in Python API and JAVA API
- internally executables handled as lists not as Sets
- python api does not allow for adding edges based on id's.
- also have a getJob function based on ID.
- escape function in python api?is it for all strings
- JAVA API *\[Gaurang\]*unmigrated-wiki-markup
- Python API *\[Gideon\]*unmigrated-wiki-markup
- Perl API *\[Jens\]*
pegasus-transfer to support pegasus-s3 tool Gideon wrote *\[Mats\]*unmigrated-wiki-markup Wiki Markup
- changes to Pegasus to use pegasus-transfer for S3 *\[Karan\]*
- handle transfer of S3 config file etc
- get s3 to work with SeqExec launcher.
- refactoring of clustered jobs (internal to pegasus)
- Addition of -conf option
Java Clients *\[Prasanth,Rajiv,Karan\]*unmigrated-wiki-markup Wiki Markup
- Python Clients *\[Prasanth,Fabio\]*
- pegasus-statistics, pegasus-plots, pegasus-analyzer, monitord
- Perl Clients *\[Gaurang\]*
- Improvements to pegasus-tc-client *\[Prasanth\]*
- the pegasus-tc-client output is in the old deprecated format.
- Improvements to pegasus-rc-client *\[Rajiv\]*
- Investigation of RLS compatibility issues
- Addition of default categories to allow for easier specification of category based knobs at DAGMan level
- cleanup jobs
- subdax jobs
- Improve Rescue DAG semantics *\[Rajiv\]*
Pegasus should not require the jobmanager compute to be present in site catalog. *\[Karan\]* [https://jira.isi.edu/browse/PM-277] Wiki Markup
Condor Common Log Handling?
Improve the Condor File IO mode in Pegasus ? Not clear how to do it without going down the staging-sites option.
User experience can be improved, but would be a hack to do it in Pegasus without staging-sites option.
- Dependant on Bill
Testing Framework and Testing
*People Involved \ [Jens,Gaurang\]*
Porting the VM to 3.1.0
*People Involved \ [Karan,Rajiv\]*
- Addition of new exercises