Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 4.0

...

  • Main JIRA Issue
  • Updated Database Schema
  • Wiki MarkupAdditional DB schema changes to be able to connect jobs/tasks in the DAX with corresponding kickstart records *\[Karan,Fabio,Monte\]* Wiki Markup
  • Identify the new events for the updated schema *\[Karan,Fabio\]*
    • Wiki MarkupPegasus to generate netlogger events to a file in the submit directory *\[Karan\]*
    • Wiki MarkupChanges to monitord to conform to new schema, populate netlogger stream from Pegasus. *\[Fabio\]*
    unmigrated-wiki-markup
  • Addition of workflow metrics file containing distribution of jobs into the DB *\[Fabio,Karan\]*
    • Wiki MarkupRevive the metrics file created by Pegasus . Should be populated in the submit directory. *\[Karan\]*

Notification Support in Pegasus via monitord

  • Main JIRA Issue unmigrated-wiki-markup
  • Monitord needs to be managed by Condor. What happens if monitord crashes or condor/system crashes? We want monitord to come up automatically as Condor recovers after a restart. *\[Fabio,Gaurang\]*
  • Wiki MarkupMonitord needs to support notifications *\[Fabio\]*unmigrated-wiki-markup
  • Requires changes to Pegasus to generate input file for monitord *\[Rajiv,Karan\]*unmigrated-wiki-markup
  • Come up with default notify scripts in the toolkit that notify the user and generate some status reports. *\[Gaurang\]*
  • Changes to DAX Schema
    • Main JIRA Item PM-350
    • Addition of invoke element at the workflow levelunmigrated-wiki-markup
    • Changes to python API *\[Gideon\]* Wiki Markup
    • Changes to Perl API *\[Jens\]*unmigrated-wiki-markup
    • Changes to JAVA API *\[Gaurang\]* Wiki Markup
    • Change to JAVA Parser *\[Karan\]*unmigrated-wiki-markup
    • Fabio needs to make sure exitcodes are thrown correctly and restarts are handled correctly. *\[Fabio\]*

...

Monitord

...

Management

...

[Fabio,Gaurang

...

]

...

https://confluence.pegasus.isi.edu/display/pegasus/Monitord+Management+via+Condor

Auxillary Tools to Stampede DB

  • Wiki Markuppegasus-statistics *\[Prasanth, Mats\]* Wiki Markup
  • pegasus-plots *\[Prasanth\]* Wiki Markup
  • pegasus-analyzer *\[Fabio\]*

...

Monitord

...

Changes

...

[Fabio

...

]

...

  • Monitord also needs to be able to account for newer versions of Condor DAGMan creating a jobstate.log file.

...

  • Feedback from Duncan brown while using Python API
  • Executable handling in Python API and JAVA API
    • internally executables handled as lists not as Sets
    • python api does not allow for adding edges based on id's.
    • also have a getJob function based on ID.
    • escape function in python api?is it for all strings
    unmigrated-wiki-markup
  • JAVA API *\[Gaurang\]*unmigrated-wiki-markup
  • Python API *\[Gideon\]* Wiki Markup
  • Perl API *\[Jens\]*

...

S3

...

support

...

[Mats,

...

Karan

...

]

...

  • Wiki Markuppegasus-transfer to support pegasus-s3 tool Gideon wrote *\[Mats\]* Wiki Markup
  • changes to Pegasus to use pegasus-transfer for S3 *\[Karan\]*
    • handle transfer of S3 config file etc
    • get s3 to work with SeqExec launcher.

...

  • refactoring of clustered jobs (internal to pegasus)
  • Addition of -conf option
    • Wiki MarkupJava Clients *\[Prasanth,Rajiv,Karan\]*unmigrated-wiki-markup
    • Python Clients *\[Prasanth,Fabio\]*
      • pegasus-statistics, pegasus-plots, pegasus-analyzer, monitord
    • Wiki MarkupPerl Clients *\[Gaurang\]*
    Wiki Markup
  • Improvements to pegasus-tc-client *\[Prasanth\]*
    • the pegasus-tc-client output is in the old deprecated format.
    unmigrated-wiki-markup
  • Improvements to pegasus-rc-client *\[Rajiv\]*
    • Investigation of RLS compatibility issues
  • Addition of default categories to allow for easier specification of category based knobs at DAGMan level
    • cleanup jobs
    • subdax jobs
    unmigrated-wiki-markup
  • Improve Rescue DAG semantics *\[Rajiv\]*
  • Wiki MarkupPegasus should not require the jobmanager compute to be present in site catalog. *\[Karan\]* [https://jira.isi.edu/browse/PM-277]

Condor Common Log Handling?

...

Improve the Condor File IO mode in Pegasus ? Not clear how to do it without going down the staging-sites option.
User experience can be improved, but would be a hack to do it in Pegasus without staging-sites option.

...

User

...

Guide

...

Reorganization

...

[Bill

...

]

  • Dependant on Bill

Testing Framework and Testing

Wiki Markup*People Involved \ [Jens,Gaurang\]*

Porting the VM to 3.1.0

...

*People Involved \ [Karan,Rajiv\]*

  • Addition of new exercises

...