The information in the kickstart output files and the condor dagman logs can be mined to retrieve useful statistics about how the workflow ran. The data retrieved then can be used to generate useful graphs. This section lists the various ways to visualize and mine statistics using helper scripts in the Pegasus distribution and remote tools.

Visualizing Graph Structure of A Workflow

Visualizing the structure of a workflow is a two step process. The first step is to convert the workflow description (DAX or DAG) into a DOT file. DOT is a special file format used for the visual display of graphs. The next step is to view the DOT file in a viewer, or use it to generate an image. These steps are described in more detail below.

Generating DOT from DAX

You can generate a DOT file from a DAX file using the pegasus-graphviz tool provided with Pegasus in the $PEGASUS_HOME/bin directory.

Usage

Usage: pegasus-graphviz [options] FILE

Parses FILE and generates a DOT-formatted graphical representation of the DAG.
FILE can be a Condor DAGMan file, or a Pegasus DAX file.

Options:

  -h, --help            show this help message and exit
  -s, --nosimplify      Do not simplify the graph by removing redundant edges.
                        [default: False]
  -l LABEL, --label=LABEL
                        What attribute to use for labels. One of 'label',
                        'xform', 'id', 'xform-id', 'label-xform', 'label-id'.
                        For 'label', the transformation is used for jobs that
                        have no node-label. [default: label]
  -o FILE, --output=FILE
                        Write output to FILE [default: stdout]
  -r XFORM, --remove=XFORM
                        Remove jobs from the workflow by transformation name
  -W WIDTH, --width=WIDTH
                        Width of the digraph
  -H HEIGHT, --height=HEIGHT
                        Height of the digraph

  -f, --files           Include files. This option is only valid for DAX
                        files. [default: false]

Example

$PEGASUS_HOME/bin/pegasus-graphviz --output scb.dot scb_dax.xml

Generating DOT from DAG

Pegasus automatically generates a DOT file (<daxlabel-daxindex>.dot) for each executable workflow and saves it in the submit directory. You can use this file, or you can generate a different one using the pegasus-graphviz tool provided with Pegasus ($PEGASUS_HOME/bin/pegasus-graphviz). The difference is that pegasus-graphviz gives you some additional options that aren't available if you use the automatically-generated DOT file.

Usage

Usage: pegasus-graphviz [options] FILE

Parses FILE and generates a DOT-formatted graphical representation of the DAG.
FILE can be a Condor DAGMan file, or a Pegasus DAX file.

Options:

  -h, --help            show this help message and exit
  -s, --nosimplify      Do not simplify the graph by removing redundant edges.
                        [default: False]
  -l LABEL, --label=LABEL
                        What attribute to use for labels. One of 'label',
                        'xform', 'id', 'xform-id', 'label-xform', 'label-id'.
                        For 'label', the transformation is used for jobs that
                        have no node-label. [default: label]
  -o FILE, --output=FILE
                        Write output to FILE [default: stdout]
  -r XFORM, --remove=XFORM
                        Remove jobs from the workflow by transformation name
  -W WIDTH, --width=WIDTH
                        Width of the digraph
  -H HEIGHT, --height=HEIGHT
                        Height of the digraph

  -f, --files           Include files. This option is only valid for DAX
                        files. [default: false]

Example

$PEGASUS_HOME/bin/pegasus-graphviz --output scb.dot scb-0.dag

Viewing DOT files

DOT files can be used to generate images or displayed using a viewer. You can find DOT file viewers at http://www.graphviz.org. In addition, on Mac OS X the OmniGraffle program can read and display DOT files. The advantage of OmniGraffle is that you can edit the DOT file visually and export it in a number of formats.

To generate a jpeg file using the "dot" program distributed with GraphViz run:

dot -Tjpeg -o SCB_DAX.jpg  scb.dot

Here is an example image generated from a DAX.

Here is an example image generated from a DAG.

Visualizing a Single Workflow Run

show-job

show-job is a perl script that can be used to generate a Gantt char of a workflow run.
It generates the gantt chart in ploticus input format and then generates an eps file and a png file using the ploticus program. The ploticus executable should be in your path.

Usage

$PEGASUS_HOME/contrib/showlog/show-job --color-file <the file mapping job transformation names to color> <path to the dag file>

Sample Usage

sukhna 59% $PEGASUS_HOME/contrib/showlog/show-job --color-file color.in dags/vahi/pegasus/scb/run0001/scb-0.dag
# min=1233364634 2009-01-30T17:17:14-08:00
# max=1233376529 2009-01-30T20:35:29-08:00
# diff=11895
# running y=1...
# xstubs=1800, xticks=600, width=13.2166666666667, height=5
job scb::das_tide has color green
job pegasus::transfer has color magenta
job pegasus::dirmanager has color lavender
job scb::fcst_tide has color orange
job pegasus::rc-client has color powderblue2
job unknown has color gray(0.75)
job scb::interpolate has color blue
# /old-usr/sukhna/install/ploticus/pl232src/bin/ploticus /tmp/sj-UDs7SZ-1.pls -eps -o /lfs1/work/jpl/dags/vahi/pegasus/scb/run0001/scb-1.eps
# /usr/ucb/convert -density 96x96 /lfs1/work/jpl/dags/vahi/pegasus/scb/run0001/scb-1.eps /lfs1/work/jpl/dags/vahi/pegasus/scb/run0001/scb-1.png
# running y=2...
# xstubs=1800, xticks=600, width=13.2166666666667, height=5
job scb::das_tide has color green
job pegasus::transfer has color magenta
job pegasus::dirmanager has color lavender
job scb::fcst_tide has color orange
job pegasus::rc-client has color powderblue2
job unknown has color gray(0.75)
job scb::interpolate has color blue
# /old-usr/sukhna/install/ploticus/pl232src/bin/ploticus /tmp/sj-UDs7SZ-2.pls -eps -o /lfs1/work/jpl/dags/vahi/pegasus/scb/run0001/scb-2.eps
# /usr/ucb/convert -density 96x96 /lfs1/work/jpl/dags/vahi/pegasus/scb/run0001/scb-2.eps /lfs1/work/jpl/dags/vahi/pegasus/scb/run0001/scb-2.png

number of jobs: 8
number of script failures: 0
sequential duration of jobs: 9090 s
total workflow duration: 11895 s (speed-up 0.8)

Sample color file
The color file can be used to provide different colors for the different transformations in the DAX.

scb::das_tide           green
scb::fcst_tide          orange
scb::interpolate        blue

Here is a sample gantt chart for a workflow execution run

SCB Workflow Gantt Chart
Added by UWC, the Universal Wiki Converter

Click on the above image to see an enlarged view

Visualizing a Workflow of Workflows Runs using Netlogger

If a user executes a workflow that in turn contains other sub workflows, it is possible to visualize them with little help from the netlogger folks.

SCEC runs a workflow of workflows, where each outer level workflow has about 80 sub workflows. The workflow logs need to be populated into the netlogger database, and then a gantt chart can be plotting using R statistical package.

Here is a sample cummulative time gantt chart

Netlogger Visualization of SCEC workflows

Click on the above image to see an enlarged view

  • No labels