Introduction

The purpose of this is to verify numbers out of a netlogger db by executing a 0.5 degree montage workflow runs for various pegasus configurations and target execution environments.
The execution is then loaded into Pegasus Provenance Tracking Catalog and also into the Netlogger DB.
Various queries are then executed against both PTC and Netlogger DB.
The results are compared to make sure the numbers match.

Pegasus Provenance Tracking Catalog

To enable population of the kickstart records into the PTC the following pegasus properties are required to be set

pegasus.catalog.provenance=InvocationSchema
pegasus.catalog.*.db.driver = MySQL
pegasus.catalog.*.db.url = jdbc:mysql://corbusier.isi.edu/pegasus_montage
pegasus.catalog.*.db.user = <dbuser>
pegasus.catalog.*.db.password = <password>

PTC Queries Per Workflow Per Job Type

Runtime Breakdown by job type per workflow

SELECT wf_label,tr_name, count(tr_name) as number, round(sum(duration),2) as sum_seconds,
{panel}
            round(sum(duration)/(3600),2) as sum_hours, round(avg(duration),2) as avg_seconds
{panel}
FROM ptc_invocation
WHERE wf_label LIKE 'condorg_nocluster_0.5'
GROUP BY tr_name;

Number of failures by job type per workflow

SELECT wf_label,tr_name, count(tr_name) as number
FROM ptc_invocation JOIN ptc_job on ptc_invocation.id=ptc_job.id
WHERE ptc_job.type='M' AND ptc_job.exitcode != 0 AND  wf_label LIKE 'condorg_nocluster_0.5'
GROUP BY tr_name;

Montage Workflow Runs

The 0.5 degree montage workflow was executed as follows on the viz cluster at ISI

  • via CondorG without clustering
  • via CondorG with horizontal clustering and a bundle factor of 4
  • via Condor glidein without clustering
  • via Condor Glidein with horizontal clustering and a bundle factor of 4

Montage Validation Runs

DAX /Abstract Workflow

Number of Jobs by Type for 0.5 degree workflow

Type of Job

Number

mProject

15

mDiffFit

29

mConcatFit

1

mBgModel

1

mBackground

15

mImgTable

1

mAdd

1

mShrink

1

mJPEG

1

Total

65

run0001 ( via CondorG without clustering )

PTC Numbers

Runtime Breakdown by job type

+-----------------------+-------------+--------+-------------+-------------+
| wf_label              | tr_name     | number | sum_seconds | avg_seconds |
+-----------------------+-------------+--------+-------------+-------------+
| condorg_nocluster_0.5 | dirmanager  |      1 |        0.10 |        0.10 |
| condorg_nocluster_0.5 | mAdd        |      1 |       11.60 |       11.60 |
| condorg_nocluster_0.5 | mBackground |     15 |      110.99 |        7.40 |
| condorg_nocluster_0.5 | mBgModel    |      1 |        0.08 |        0.08 |
| condorg_nocluster_0.5 | mConcatFit  |      1 |        0.11 |        0.11 |
| condorg_nocluster_0.5 | mDiffFit    |     29 |       83.61 |        2.88 |
| condorg_nocluster_0.5 | mImgtbl     |      1 |        0.37 |        0.37 |
| condorg_nocluster_0.5 | mJPEG       |      1 |        1.01 |        1.01 |
| condorg_nocluster_0.5 | mProjectPP  |     15 |      162.38 |       10.83 |
| condorg_nocluster_0.5 | mShrink     |      1 |        2.26 |        2.26 |
| condorg_nocluster_0.5 | rc-client   |      3 |        0.72 |        0.24 |
| condorg_nocluster_0.5 | transfer    |      4 |       20.19 |        5.05 |
+-----------------------+-------------+--------+-------------+-------------+

Number of failures by job type

Zero Failures

Number of successful jobs by job type

+-----------------------+-------------+--------+
| wf_label              | tr_name     | number |
+-----------------------+-------------+--------+
| condorg_nocluster_0.5 | dirmanager  |      1 |
| condorg_nocluster_0.5 | mAdd        |      1 |
| condorg_nocluster_0.5 | mBackground |     15 |
| condorg_nocluster_0.5 | mBgModel    |      1 |
| condorg_nocluster_0.5 | mConcatFit  |      1 |
| condorg_nocluster_0.5 | mDiffFit    |     29 |
| condorg_nocluster_0.5 | mImgtbl     |      1 |
| condorg_nocluster_0.5 | mJPEG       |      1 |
| condorg_nocluster_0.5 | mProjectPP  |     15 |
| condorg_nocluster_0.5 | mShrink     |      1 |
| condorg_nocluster_0.5 | rc-client   |      3 |
| condorg_nocluster_0.5 | transfer    |      4 |
+-----------------------+-------------+--------+

Netlogger Numbers

To be filled by Chris

Runtime Breakdown by job type

job name

number

total time

average time

mBgModel:3.0

1

0.085

0.085

pegasus::dirmanager

1

0.104

0.104

mBackground:3.0

15

110.991

7.399

mProjectPP:3.0

15

162.377

10.825

pegasus::transfer

4

20.189

5.048

pegasus::rc-client

3

0.722

0.241

mDiffFit:3.0

29

83.606

2.883

mImgtbl:3.0

1

0.368

0.368

mAdd:3.0

1

11.605

11.605

mConcatFit:3.0

1

0.107

0.107

mShrink:3.0

1

2.265

2.265

mJPEG:3.0

1

1.009

1.009

Number of failures by job type

no failures

Number of successful jobs by job type

job name

number

mBgModel:3.0

1

pegasus::dirmanager

1

mBackground:3.0

15

mProjectPP:3.0

15

pegasus::transfer

4

pegasus::rc-client

3

mDiffFit:3.0

29

mImgtbl:3.0

1

mAdd:3.0

1

mConcatFit:3.0

1

mShrink:3.0

1

mJPEG:3.0

1

run0002 ( via CondorG with clustering )

PTC Numbers

Runtime Breakdown by job type

+---------------------+-------------+--------+-------------+-------------+
| wf_label            | tr_name     | number | sum_seconds | avg_seconds |
+---------------------+-------------+--------+-------------+-------------+
| condorg_cluster_0.5 | dirmanager  |      1 |        0.07 |        0.07 |
| condorg_cluster_0.5 | mAdd        |      1 |       13.11 |       13.11 |
| condorg_cluster_0.5 | mBackground |     15 |       75.20 |        5.01 |
| condorg_cluster_0.5 | mBgModel    |      1 |        0.11 |        0.11 |
| condorg_cluster_0.5 | mConcatFit  |      1 |        0.11 |        0.11 |
| condorg_cluster_0.5 | mDiffFit    |     29 |       70.65 |        2.44 |
| condorg_cluster_0.5 | mImgtbl     |      1 |        0.33 |        0.33 |
| condorg_cluster_0.5 | mJPEG       |      1 |        0.97 |        0.97 |
| condorg_cluster_0.5 | mProjectPP  |     15 |       97.37 |        6.49 |
| condorg_cluster_0.5 | mShrink     |      1 |        2.26 |        2.26 |
| condorg_cluster_0.5 | rc-client   |      3 |        0.73 |        0.24 |
| condorg_cluster_0.5 | transfer    |      4 |       19.00 |        4.75 |
+---------------------+-------------+--------+-------------+-------------+

Number of failures by job type

Zero Failures

Number of successful jobs by job type

+---------------------+-------------+--------+
| wf_label            | tr_name     | number |
+---------------------+-------------+--------+
| condorg_cluster_0.5 | dirmanager  |      1 |
| condorg_cluster_0.5 | mAdd        |      1 |
| condorg_cluster_0.5 | mBackground |     15 |
| condorg_cluster_0.5 | mBgModel    |      1 |
| condorg_cluster_0.5 | mConcatFit  |      1 |
| condorg_cluster_0.5 | mDiffFit    |     29 |
| condorg_cluster_0.5 | mImgtbl     |      1 |
| condorg_cluster_0.5 | mJPEG       |      1 |
| condorg_cluster_0.5 | mProjectPP  |     15 |
| condorg_cluster_0.5 | mShrink     |      1 |
| condorg_cluster_0.5 | rc-client   |      3 |
| condorg_cluster_0.5 | transfer    |      4 |
+---------------------+-------------+--------+

Netlogger Numbers

To be filled by Chris

Runtime Breakdown by job type

name

number

total time

average time

mBgModel:3.0

1

0.112

0.112

mProjectPP:3.0

15

97.373

6.4915

mBackground:3.0

15

75.198

5.0132

pegasus::dirmanager

1

0.072

0.072

pegasus::transfer

4

19.003

4.751

pegasus::rc-client

3

0.733

0.244

mDiffFit:3.0

29

70.647

2.436

mImgtbl:3.0

1

0.329

0.329

mAdd:3.0

1

13.107

13.107

mConcatFit:3.0

1

0.107

0.107

mShrink:3.0

1

2.258

2.258

mJPEG:3.0

1

0.97

0.97

Number of failures by job type

no failures

Number of successful jobs by job type

job name

number

mBgModel:3.0

1

pegasus::dirmanager

1

mBackground:3.0

15

mProjectPP:3.0

15

pegasus::transfer

4

pegasus::rc-client

3

mDiffFit:3.0

29

mImgtbl:3.0

1

mAdd:3.0

1

mConcatFit:3.0

1

mShrink:3.0

1

mJPEG:3.0

1

run0003 ( via Condor Glidein without clustering )

PTC Numbers

Runtime Breakdown by job type

+-----------------------+-------------+--------+-------------+-------------+
| wf_label              | tr_name     | number | sum_seconds | avg_seconds |
+-----------------------+-------------+--------+-------------+-------------+
| glidein_nocluster_0.5 | dirmanager  |      1 |        0.14 |        0.14 |
| glidein_nocluster_0.5 | mAdd        |      1 |       11.43 |       11.43 |
| glidein_nocluster_0.5 | mBackground |     15 |       34.63 |        2.31 |
| glidein_nocluster_0.5 | mBgModel    |      1 |        0.11 |        0.11 |
| glidein_nocluster_0.5 | mConcatFit  |      1 |        0.07 |        0.07 |
| glidein_nocluster_0.5 | mDiffFit    |     29 |       49.40 |        1.70 |
| glidein_nocluster_0.5 | mImgtbl     |      1 |        0.53 |        0.53 |
| glidein_nocluster_0.5 | mJPEG       |      1 |        1.01 |        1.01 |
| glidein_nocluster_0.5 | mProjectPP  |     15 |       72.61 |        4.84 |
| glidein_nocluster_0.5 | mShrink     |      1 |        3.58 |        3.58 |
| glidein_nocluster_0.5 | rc-client   |      9 |        2.19 |        0.24 |
| glidein_nocluster_0.5 | transfer    |      4 |       19.64 |        4.91 |
+-----------------------+-------------+--------+-------------+-------------+

Number of failures by job type

+-----------------------+-----------+--------+
| wf_label              | tr_name   | number |
+-----------------------+-----------+--------+
| glidein_nocluster_0.5 | rc-client |      9 |
+-----------------------+-----------+--------+

Number of successful jobs by job type

+-----------------------+-------------+--------+
| wf_label              | tr_name     | number |
+-----------------------+-------------+--------+
| glidein_nocluster_0.5 | dirmanager  |      1 |
| glidein_nocluster_0.5 | mAdd        |      1 |
| glidein_nocluster_0.5 | mBackground |     15 |
| glidein_nocluster_0.5 | mBgModel    |      1 |
| glidein_nocluster_0.5 | mConcatFit  |      1 |
| glidein_nocluster_0.5 | mDiffFit    |     29 |
| glidein_nocluster_0.5 | mImgtbl     |      1 |
| glidein_nocluster_0.5 | mJPEG       |      1 |
| glidein_nocluster_0.5 | mProjectPP  |     15 |
| glidein_nocluster_0.5 | mShrink     |      1 |
| glidein_nocluster_0.5 | transfer    |      4 |
+-----------------------+-------------+--------+

Netlogger Numbers

To be filled by Chris

Runtime Breakdown by job type

job name

number

total time

average time

mAdd:3.0

1

11.427

11.427

mBackground:3.0

15

34.626

2.3084

mBgModel:3.0

1

0.11

0.11

mConcatFit:3.0

1

0.071

0.071

mDiffFit:3.0

29

49.4

1.703

mImgtbl:3.0

1

0.526

0.526

mJPEG:3.0

1

1.008

1.008

mProjectPP:3.0

15

72.61

4.840

mShrink:3.0

1

3.576

3.576

pegasus::dirmanager

1

0.143

0.143

pegasus::rc-client

9

2.186

0.243

pegasus::transfer

4

19.64

4.91

Number of failures by job type

job

number

pegasus::rc-client

9

Number of successful jobs by job type

job name

number

mAdd:3.0

1

mBackground:3.0

15

mBgModel:3.0

1

mConcatFit:3.0

1

mDiffFit:3.0

29

mImgtbl:3.0

1

mJPEG:3.0

1

mProjectPP:3.0

15

mShrink:3.0

1

pegasus::dirmanager

1

pegasus::transfer

4

run0004 ( via Condor glidein with clustering )

PTC Numbers

Runtime Breakdown by job type

+---------------------+-------------+--------+-------------+-------------+
| wf_label            | tr_name     | number | sum_seconds | avg_seconds |
+---------------------+-------------+--------+-------------+-------------+
| glidein_cluster_0.5 | dirmanager  |      1 |        0.12 |        0.12 |
| glidein_cluster_0.5 | mAdd        |      1 |       12.92 |       12.92 |
| glidein_cluster_0.5 | mBackground |     15 |       60.55 |        4.04 |
| glidein_cluster_0.5 | mBgModel    |      1 |        0.08 |        0.08 |
| glidein_cluster_0.5 | mConcatFit  |      1 |        0.12 |        0.12 |
| glidein_cluster_0.5 | mDiffFit    |     29 |       47.72 |        1.65 |
| glidein_cluster_0.5 | mImgtbl     |      1 |        0.66 |        0.66 |
| glidein_cluster_0.5 | mJPEG       |      1 |        3.47 |        3.47 |
| glidein_cluster_0.5 | mProjectPP  |     15 |       72.22 |        4.81 |
| glidein_cluster_0.5 | mShrink     |      1 |        3.04 |        3.04 |
| glidein_cluster_0.5 | rc-client   |      9 |        2.20 |        0.24 |
| glidein_cluster_0.5 | transfer    |      4 |       18.18 |        4.54 |
+---------------------+-------------+--------+-------------+-------------+

Number of failures by job type

+---------------------+-----------+--------+
| wf_label            | tr_name   | number |
+---------------------+-----------+--------+
| glidein_cluster_0.5 | rc-client |      9 |
+---------------------+-----------+--------+

Number of successful jobs by job type

+---------------------+-------------+--------+
| wf_label            | tr_name     | number |
+---------------------+-------------+--------+
| glidein_cluster_0.5 | dirmanager  |      1 |
| glidein_cluster_0.5 | mAdd        |      1 |
| glidein_cluster_0.5 | mBackground |     15 |
| glidein_cluster_0.5 | mBgModel    |      1 |
| glidein_cluster_0.5 | mConcatFit  |      1 |
| glidein_cluster_0.5 | mDiffFit    |     29 |
| glidein_cluster_0.5 | mImgtbl     |      1 |
| glidein_cluster_0.5 | mJPEG       |      1 |
| glidein_cluster_0.5 | mProjectPP  |     15 |
| glidein_cluster_0.5 | mShrink     |      1 |
| glidein_cluster_0.5 | transfer    |      4 |
+---------------------+-------------+--------+

Netlogger Numbers

To be filled by Chris

Runtime Breakdown by job type

job name

number

total time

average time

mAdd:3.0

1

12.919

12.919

mBackground:3.0

15

60.552

4.0368

mBgModel:3.0

1

0.077

0.077

mConcatFit:3.0

1

0.12

0.12

mDiffFit:3.0

29

47.716

1.645

mImgtbl:3.0

1

0.663

0.663

mJPEG:3.0

1

3.469

3.469

mProjectPP:3.0

15

72.22

4.814

mShrink:3.0

1

3.045

3.045

pegasus::dirmanager

1

0.125

0.125

pegasus::transfer

4

18.176

4.544

pegasus::rc-client

9

2.202

0.245

Number of failures by job type

job

number

pegasus::rc-client

9

Number of successful jobs by job type

job name

number

mAdd:3.0

1

mBackground:3.0

15

mBgModel:3.0

1

mConcatFit:3.0

1

mDiffFit:3.0

29

mImgtbl:3.0

1

mJPEG:3.0

1

mProjectPP:3.0

15

mShrink:3.0

1

pegasus::dirmanager

1

pegasus::transfer

4

  • No labels