Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin

...

  • Add notification support to Pegasus
  • Moving auxillary tools to stampede db
  • Other Stampede Related Changes
  • Usability Changes
  • User Guide Reorganization 
  • Testing Framework and Testing.

JIRA Task Board

https://jira.isi.edu/secure/TaskBoard.jspa?selectedBoardId=10054&start=0&pageType=TaskBoard&type=TB&selectedProjectId=10002TaskBoard for 3.1

Items for April End

Stampede Related Changes

  • Main JIRA Issue
  • Updated Database Schema Wiki MarkupAdditional DB schema changes to be able to connect jobs/tasks in the DAX with corresponding kickstart records *\
  • Additional DB schema changes to be able to connect jobs/tasks in the DAX with corresponding kickstart records [Karan,Fabio,Monte\]*unmigrated-wiki-markup
  • Identify the new events for the updated schema *\[Karan,Fabio\]*
      unmigrated-wiki-markup
    • Pegasus to generate netlogger events to a file in the submit directory *\[Karan\]*unmigrated-wiki-markup
    • Changes to monitord to conform to new schema, populate netlogger stream from Pegasus. *\[Fabio\]*
    Wiki Markup
  • Addition of workflow metrics file containing distribution of jobs into the DB *\[Fabio,Karan\]*
      unmigrated-wiki-markup
    • Revive the metrics file created by Pegasus . Should be populated in the submit directory. *\[Karan\]*

Notification Support in Pegasus via monitord

  • Main JIRA Issue
  • Wiki MarkupMonitord needs to be managed by Condor. What happens if monitord crashes or condor/system crashes? We want monitord to come up automatically as Condor recovers after a restart. *\[Fabio,Gaurang\]*unmigrated-wiki-markup
  • Monitord needs to support notifications *\[Fabio\]*
  • Wiki MarkupRequires changes to Pegasus to generate input file for monitord *\[Rajiv,Karan\]*unmigrated-wiki-markup
  • Come up with default notify scripts in the toolkit that notify the user and generate some status reports. *\[Gaurang\]*
  • Changes to DAX Schema
    • Main JIRA Item PM-350
    • Addition of invoke element at the workflow level
    • Wiki MarkupChanges to python API *\[Gideon\]* Wiki Markup
    • Changes to Perl API *\[Jens\]*unmigrated-wiki-markup
    • Changes to JAVA API *\[Gaurang\]*unmigrated-wiki-markup
    • Change to JAVA Parser *\[Karan\]*
    • Wiki MarkupFabio needs to make sure exitcodes are thrown correctly and restarts are handled correctly. *\[Fabio\]*

...

Monitord

...

Management

...

[Fabio,Gaurang

...

]

...

https://confluence.pegasus.isi.edu/display/pegasus/Monitord+Management+via+Condor

Auxillary Tools to Stampede DB

...

  • pegasus-statistics *\[Prasanth, Mats\]*unmigrated-wiki-markup
  • pegasus-plots *\[Prasanth\]*unmigrated-wiki-markup
  • pegasus-analyzer *\[Fabio\]*

...

Monitord

...

Changes

...

[Fabio

...

]

...

  • Monitord also needs to be able to account for newer versions of Condor DAGMan creating a jobstate.log file.

...

  • Feedback from Duncan brown while using Python API
  • Executable handling in Python API and JAVA API
    • internally executables handled as lists not as Sets
    • python api does not allow for adding edges based on id's.
    • also have a getJob function based on ID.
    • escape function in python api?is it for all strings
    unmigrated-wiki-markup
  • JAVA API *\[Gaurang\]*unmigrated-wiki-markup
  • Python API *\[Gideon\]*unmigrated-wiki-markup
  • Perl API *\[Jens\]*

...

S3

...

support

...

[Mats,

...

Karan

...

]

...

  • Wiki Markuppegasus-transfer to support pegasus-s3 tool Gideon wrote *\[Mats\]*unmigrated-wiki-markup
  • changes to Pegasus to use pegasus-transfer for S3 *\[Karan\]*
    • handle transfer of S3 config file etc
    • get s3 to work with SeqExec launcher.

...

  • refactoring of clustered jobs (internal to pegasus)
  • Addition of -conf option
    • Wiki MarkupJava Clients *\[Prasanth,Rajiv,Karan\]*unmigrated-wiki-markup
    • Python Clients *\[Prasanth,Fabio\]*
      • pegasus-statistics, pegasus-plots, pegasus-analyzer, monitord
      unmigrated-wiki-markup
    • Perl Clients *\[Gaurang\]*
    unmigrated-wiki-markup
  • Improvements to pegasus-tc-client *\[Prasanth\]*
    • the pegasus-tc-client output is in the old deprecated format.
    unmigrated-wiki-markup
  • Improvements to pegasus-rc-client *\[Rajiv\]*
    • Investigation of RLS compatibility issues
  • Addition of default categories to allow for easier specification of category based knobs at DAGMan level
    • cleanup jobs
    • subdax jobs
    unmigrated-wiki-markup
  • Improve Rescue DAG semantics *\[Rajiv\]*
  • Wiki MarkupPegasus should not require the jobmanager compute to be present in site catalog. *\[Karan\]* [https://jira.isi.edu/browse/PM-277]

Condor Common Log Handling?

...

Improve the Condor File IO mode in Pegasus ? Not clear how to do it without going down the staging-sites option.
User experience can be improved, but would be a hack to do it in Pegasus without staging-sites option.

...

User

...

Guide

...

Reorganization

...

[Bill

...

]

  • Dependant on Bill

Testing Framework and Testing

...

*People Involved \ [Jens,Gaurang\]*

Porting the VM to 3.1.0

...

*People Involved \ [Karan,Rajiv\]*

  • Addition of new exercises

...