The information in the kickstart output files and the condor dagman logs can be mined to retrieve useful statistics about how the workflow ran. The data retrieved then can be used to generate useful graphs. This section lists the various ways to visualize and mine statistics using helper scripts in the Pegasus distribution and remote tools.
Visualizing Graph Structure of A Workflow
Visualizing the structure of a workflow is a two step process. The first step is to convert the workflow description (DAX or DAG) into a DOT file. DOT is a special file format used for the visual display of graphs. The next step is to view the DOT file in a viewer, or use it to generate an image. These steps are described in more detail below.
Generating DOT from DAX
You can generate a DOT file from a DAX file using the pegasus-graphviz tool provided with Pegasus in the $PEGASUS_HOME/bin directory.
Usage
Usage: pegasus-graphviz [options] FILE
Parses FILE and generates a DOT-formatted graphical representation of the DAG.
FILE can be a Condor DAGMan file, or a Pegasus DAX file.
Options:
-h, --help show this help message and exit
-s, --nosimplify Do not simplify the graph by removing redundant edges.
[default: False]
-l LABEL, --label=LABEL
What attribute to use for labels. One of 'label',
'xform', 'id', 'xform-id', 'label-xform', 'label-id'.
For 'label', the transformation is used for jobs that
have no node-label. [default: label]
-o FILE, --output=FILE
Write output to FILE [default: stdout]
-r XFORM, --remove=XFORM
Remove jobs from the workflow by transformation name
-W WIDTH, --width=WIDTH
Width of the digraph
-H HEIGHT, --height=HEIGHT
Height of the digraph
-f, --files Include files. This option is only valid for DAX
files. [default: false]
Example
$PEGASUS_HOME/bin/pegasus-graphviz --output scb.dot scb_dax.xml
Generating DOT from DAG
Pegasus automatically generates a DOT file (<daxlabel-daxindex>.dot) for each executable workflow and saves it in the submit directory. You can use this file, or you can generate a different one using the pegasus-graphviz tool provided with Pegasus ($PEGASUS_HOME/bin/pegasus-graphviz). The difference is that pegasus-graphviz gives you some additional options that aren't available if you use the automatically-generated DOT file.
Usage
Usage: pegasus-graphviz [options] FILE
Parses FILE and generates a DOT-formatted graphical representation of the DAG.
FILE can be a Condor DAGMan file, or a Pegasus DAX file.
Options:
-h, --help show this help message and exit
-s, --nosimplify Do not simplify the graph by removing redundant edges.
[default: False]
-l LABEL, --label=LABEL
What attribute to use for labels. One of 'label',
'xform', 'id', 'xform-id', 'label-xform', 'label-id'.
For 'label', the transformation is used for jobs that
have no node-label. [default: label]
-o FILE, --output=FILE
Write output to FILE [default: stdout]
-r XFORM, --remove=XFORM
Remove jobs from the workflow by transformation name
-W WIDTH, --width=WIDTH
Width of the digraph
-H HEIGHT, --height=HEIGHT
Height of the digraph
-f, --files Include files. This option is only valid for DAX
files. [default: false]
Example
$PEGASUS_HOME/bin/pegasus-graphviz --output scb.dot scb-0.dag
Viewing DOT files
DOT files can be used to generate images or displayed using a viewer. You can find DOT file viewers at http://www.graphviz.org. In addition, on Mac OS X the OmniGraffle program can read and display DOT files. The advantage of OmniGraffle is that you can edit the DOT file visually and export it in a number of formats.
To generate a jpeg file using the "dot" program distributed with GraphViz run:
dot -Tjpeg -o SCB_DAX.jpg scb.dot
Here is an example image generated from a DAX.
Here is an example image generated from a DAG.
Visualizing a Single Workflow Run
show-job
show-job is a perl script that can be used to generate a Gantt char of a workflow run.
It generates the gantt chart in ploticus input format and then generates an eps file and a png file using the ploticus program. The ploticus executable should be in your path.
Usage
$PEGASUS_HOME/contrib/showlog/show-job --color-file <the file mapping job transformation names to color> <path to the dag file>
Sample Usage
sukhna 59% $PEGASUS_HOME/contrib/showlog/show-job --color-file color.in dags/vahi/pegasus/scb/run0001/scb-0.dag # min=1233364634 2009-01-30T17:17:14-08:00 # max=1233376529 2009-01-30T20:35:29-08:00 # diff=11895 # running y=1... # xstubs=1800, xticks=600, width=13.2166666666667, height=5 job scb::das_tide has color green job pegasus::transfer has color magenta job pegasus::dirmanager has color lavender job scb::fcst_tide has color orange job pegasus::rc-client has color powderblue2 job unknown has color gray(0.75) job scb::interpolate has color blue # /old-usr/sukhna/install/ploticus/pl232src/bin/ploticus /tmp/sj-UDs7SZ-1.pls -eps -o /lfs1/work/jpl/dags/vahi/pegasus/scb/run0001/scb-1.eps # /usr/ucb/convert -density 96x96 /lfs1/work/jpl/dags/vahi/pegasus/scb/run0001/scb-1.eps /lfs1/work/jpl/dags/vahi/pegasus/scb/run0001/scb-1.png # running y=2... # xstubs=1800, xticks=600, width=13.2166666666667, height=5 job scb::das_tide has color green job pegasus::transfer has color magenta job pegasus::dirmanager has color lavender job scb::fcst_tide has color orange job pegasus::rc-client has color powderblue2 job unknown has color gray(0.75) job scb::interpolate has color blue # /old-usr/sukhna/install/ploticus/pl232src/bin/ploticus /tmp/sj-UDs7SZ-2.pls -eps -o /lfs1/work/jpl/dags/vahi/pegasus/scb/run0001/scb-2.eps # /usr/ucb/convert -density 96x96 /lfs1/work/jpl/dags/vahi/pegasus/scb/run0001/scb-2.eps /lfs1/work/jpl/dags/vahi/pegasus/scb/run0001/scb-2.png number of jobs: 8 number of script failures: 0 sequential duration of jobs: 9090 s total workflow duration: 11895 s (speed-up 0.8)
Sample color file
The color file can be used to provide different colors for the different transformations in the DAX.
scb::das_tide green scb::fcst_tide orange scb::interpolate blue
Here is a sample gantt chart for a workflow execution run
SCB Workflow Gantt Chart |
|---|
|
|
Added by UWC, the Universal Wiki Converter
|
Click on the above image to see an enlarged view
Visualizing a Workflow of Workflows Runs using Netlogger
If a user executes a workflow that in turn contains other sub workflows, it is possible to visualize them with little help from the netlogger folks.
SCEC runs a workflow of workflows, where each outer level workflow has about 80 sub workflows. The workflow logs need to be populated into the netlogger database, and then a gantt chart can be plotting using R statistical package.
Here is a sample cummulative time gantt chart
Netlogger Visualization of SCEC workflows |
|---|
|
|
|
Click on the above image to see an enlarged view