Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin
Panel

Table Of Contents:

Table of Contents
maxLevel5
outlinetrue
stylenone

Last Updated March 25th April 5th , 2011

Main Focus

  • Add notification support to Pegasus
  • Moving auxillary tools to stampede db
  • Other Stampede Related Changes
  • Usability Changes
  • User Guide Reorganization 
  • Testing Framework and Testing.

JIRA Task Board

TaskBoard for 3.1

Items for April End

Stampede Related Changes

  • Main JIRA Issue
  • Updated Database Schema
  • Additional DB schema changes to be able to connect jobs/tasks in the DAX with corresponding kickstart records [Karan,Fabio,Monte]
  • Identify the new events for the updated schema [Karan,Fabio]
    • Pegasus to generate netlogger events to a file in the submit directory [Karan]
    • Changes to monitord to conform to new schema, populate netlogger stream from Pegasus. [Fabio]
  • Addition of workflow metrics file containing distribution of jobs into the DB [Fabio,Karan]
    • Revive the metrics file created by Pegasus . Should be populated in the submit directory. [Karan]

Notification Support in Pegasus via monitord

  • unmigrated-wiki-markupMain JIRA Issue
  • Monitord needs to be managed by Condor. What happens if monitord crashes or condor/system crashes? We want monitord to come up automatically as Condor recovers after a restart. *\[Fabio,Gaurang\]*\*
  • Wiki MarkupMonitord needs to support notifications *\[Fabio\]*
  • Wiki MarkupRequires changes to Pegasus to generate input file for monitord *\[Rajiv,Karan\]*
  • Wiki MarkupCome up with default notify scripts in the toolkit that notify the user and generate some status reports. *\[Gaurang\]*
  • Changes to DAX Schema
    • Main JIRA Item PM-350
    • Addition of invoke element at the workflow level
    • Wiki MarkupChanges to python API *\[Gideon\]*unmigrated-wiki-markup
    • Changes to Perl API *\[Jens\]*unmigrated-wiki-markup
    • Changes to JAVA API *\[Gaurang\]* Wiki Markup
    • Change to JAVA Parser *\[Karan\]*
    • Wiki Markup
      Instead of pegasus-run launching monitord, monitord should appear as an independent job in the workflow with the highest priority *\[Karan\]*
    • Wiki MarkupFabio needs to make sure exitcodes are thrown correctly and restarts are handled correctly. *\[Fabio\]*

...

Monitord

...

Management

...

[Fabio,Gaurang

...

]

...

https://confluence.pegasus.isi.edu/display/pegasus/Monitord+Management+via+Condor

Auxillary Tools to Stampede DB

  • Wiki Markuppegasus-statistics *\[Prasanth\]*unmigrated-wiki-markup, Mats]
  • pegasus-plots *\[Prasanth\]* Wiki Markup
  • pegasus-analyzer *\[Fabio\]*

...

Monitord

...

Changes

...

[Fabio

...

]

...

  • Monitord also needs to be able to account for newer versions of Condor DAGMan creating a jobstate.log file.

Stampede Related Changes

...

Wiki Markup
Additional DB schema changes to be able to connect jobs/tasks in the DAX with corresponding kickstart records *\[Fabio,Monte\]*
Wiki Markup
Addition of workflow metrics file containing distribution of jobs into the DB *\[Fabio,Karan\]*

...

Wiki Markup
Revive the metrics file created by Pegasus . Should be populated in the submit directory. *\[Karan\]*

...

DAX API changes

  • Feedback from Duncan brown while using Python API
  • Executable handling in Python API and JAVA API
    • internally executables handled as lists not as Sets
    • python api does not allow for adding edges based on id's.
    • also have a getJob function based on ID.
    • escape function in python api?is it for all strings
  • JAVA API [Gaurang]
  • Python API [Gideon]
  • Perl API [Jens]

S3 support [Mats, Karan]

  • pegasus-transfer to support pegasus-s3 tool Gideon wrote [Mats]
  • changes to Pegasus to use pegasus-transfer for S3 [Karan]
    • handle transfer of S3 config file etc
    • get s3 to work with SeqExec launcher.

Usability Changes

  • refactoring of clustered jobs (internal to pegasus)
  • Addition of -conf option
      unmigrated-wiki-markup
    • Java Clients *\[Prasanth,Rajiv,Karan\]* Wiki Markup
    • Python Clients *\[Prasanth,Fabio\]*
      • pegasus-statistics, pegasus-plots, pegasus-analyzer, monitord
      unmigrated-wiki-markup
    • Perl Clients *\[Gaurang\]*
    Wiki Markup
  • Improvements to pegasus-tc-client *\[Prasanth\]*
    • the pegasus-tc-client output is in the old deprecated format.
    Wiki Markup
  • Improvements to pegasus-rc-client *\[Rajiv\]*
    • Investigation of RLS compatibility issues
  • Addition of default categories to allow for easier specification of category based knobs at DAGMan level
    • cleanup jobs
    • subdax jobs
  • Wiki MarkupImprove Rescue DAG semantics *\[Rajiv\]* [Rajiv]
  • Pegasus should not require the jobmanager compute to be present in site catalog. [Karan] https://jira.isi.edu/browse/PM-277

Condor Common Log Handling?

Open Question

Improve the Condor File IO mode in Pegasus ? Not clear how to do it without going down the staging-sites option.
User experience can be improved, but would be a hack to do it in Pegasus without staging-sites option.

...

User

...

Guide

...

Reorganization

...

[Bill

...

]

  • Dependant on Bill

Testing Framework and Testing

...

*People Involved \ [Jens,Gaurang\]*

Porting the VM to 3.1.0

Wiki Markup*People Involved \ [Karan,Rajiv\]*

  • Addition of new exercises

...

  • Fabio gets stampede related changes done end of april.
  • Fabio gets whole of may to put in notification support.
  • End of May we may have a first beta, where everything idenitifed above is done.
  • June is spent testing on the release .