You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 16 Next »

Table Of Contents:

Last Updated March 17th , 2011

Main Focus

  • Improving how we get data to worker nodes. Being able to retrieve data on the worker nodes from various sources.
  • Add notification support to Pegasus
  • Moving auxillary tools to stampede db
  • Other Stampede Related Changes
  • User Guide Reorganization 
  • Testing Framework and Testing.

Data to Worker Nodes

The main change for 3.1. The developers document has complete details
Download Developers Document
People Involved [Mats,Karan,Gaurang]

  • Refactor how clustered jobs are handled [Karan]

  • Addition of staging-sites option to pegasus-plan [Karan]

  • New Shell Job Wrapper for jobs when running on worker nodes [Karan,Mats]

  • Changes to pegasus-cleanup
    • Creation of pegasus-cleanup jobs [Karan]

    • new pegasus-cleanup client that runs locally [Mats]

  • Changes to pegasus-createdir
    • New pegasus-createdir client [Mats]

    • New java createdir client [Gaurang]

  • Special S3 support into pegasus-transfer, pegasus-cleanup and pegasus create dir [Mats]

  • Bypassing of staging-site while staging in input data [Karan]

  • Bypassing of staging-site while staging out output data [Karan]

  • Transfer of braindump file to remote workflow execution directories [Karan]

Notification Support in Pegasus via monitord

  • Monitord needs to support notifications [Fabio]

  • Requires changes to Pegasus to generate input file for monitord [Rajiv,Karan]

  • Come up with default notify scripts in the toolkit that notify the user and generate some status reports. [Gaurang]

Open Question

  • Monitord needs to be managed. What happens if monitord crashes or condor/system crashes? We want monitord to come up automatically as Condor recovers after a restart.
    • Instead of pegasus-run launching monitord, monitord should appear as an independent job in the workflow with the highest priority [Karan]

Auxillary Tools to Stampede DB

  • pegasus-statistics [Prasanth]

  • pegasus-plots [Prasanth]

  • pegasus-analyzer [Fabio]

Monitord Changes [Fabio]

  • Monitord also needs to be able to account for newer versions of Condor DAGMan creating a jobstate.log file.

Stampede Related Changes

  • Improve Rescue DAG semantics [Rajiv]

  • Additional DB schema changes to be able to connect jobs/tasks in the DAX with corresponding kickstart records [Fabio,Monte,Karan]

  • Addition of workflow metrics file containing distribution of jobs into the DB [Fabio,Karan]

    • Revive the metrics file created by Pegasus . Should be populated in the submit directory. [Karan]

    • Monitord picks up the metrics file and stores it in the DB [Fabio]

User Guide Reorganization [Bill]

  • Dependant on Bill

Testing Framework and Testing

  • No labels