Table Of Contents:
Last Updated Feb 28th , 2011
Main Focus
- Improving how we get data to worker nodes. Being able to retrieve data on the worker nodes from various sources.
- Add notification support to Pegasus
- Moving auxillary tools to stampede db
- Other Stampede Related Changes
- User Guide Reorganization ? Dependant on Bill.
Data to Worker Nodes
The main change for 3.1. The developers document has complete details
- Refactor how clustered jobs are handled
- Addition of staging-sites option to pegasus-plan
- New Shell Job Wrapper for jobs when running on worker nodes
- Changes to pegasus-cleanup
- Changes to pegasus-createdir
- Special S3 support into pegasus-transfer, pegasus-cleanup and pegasus create dir
- Bypassing of staging-site while staging in input data
- Bypassing of staging-site while staging out output data
- Transfer of braindump file to remote workflow execution directories
Notification Support in Pegasus
- Monitord needs to support notifications
- Requires changes to Pegasus to generate input file for monitord
- Come up with default notify scripts in the toolkit that notify the user and generate some status reports.
Auxillary Tools to Stampede DB
- pegasus-statistics
- pegasus-plots
- pegasus-analyzer
Stampede Related Changes
- Improve Rescue DAG semantics
- Additional DB schema changes to be able to connect jobs/tasks in the DAX with corresponding kickstart records
- Addition of workflow metrics file distribution of jobs into the DB