Introduction
The purpose of this is to verify numbers out of a netlogger db by executing a 0.5 degree montage workflow runs for various pegasus configurations and target execution environments.
The execution is then loaded into Pegasus Provenance Tracking Catalog and also into the Netlogger DB.
Various queries are then executed against both PTC and Netlogger DB.
The results are compared to make sure the numbers match.
Pegasus Provenance Tracking Catalog
To enable population of the kickstart records into the PTC the following pegasus properties are required to be set
pegasus.catalog.provenance=InvocationSchema pegasus.catalog.*.db.driver = MySQL pegasus.catalog.*.db.url = jdbc:mysql://corbusier.isi.edu/pegasus_montage pegasus.catalog.*.db.user = <dbuser> pegasus.catalog.*.db.password = <password>
PTC Queries Per Workflow Per Job Type
Runtime Breakdown by job type per workflow
SELECT wf_label,tr_name, count(tr_name) as number, round(sum(duration),2) as sum_seconds,
{panel}
round(sum(duration)/(3600),2) as sum_hours, round(avg(duration),2) as avg_seconds
{panel}
FROM ptc_invocation
WHERE wf_label LIKE 'condorg_nocluster_0.5'
GROUP BY tr_name;
Number of failures by job type per workflow
SELECT wf_label,tr_name, count(tr_name) as number FROM ptc_invocation JOIN ptc_job on ptc_invocation.id=ptc_job.id WHERE ptc_job.type='M' AND ptc_job.exitcode != 0 AND wf_label LIKE 'condorg_nocluster_0.5' GROUP BY tr_name;
Montage Workflow Runs
The 0.5 degree montage workflow was executed as follows on the viz cluster at ISI
- via CondorG without clustering
- via CondorG with horizontal clustering and a bundle factor of 4
- via Condor glidein without clustering
- via Condor Glidein with horizontal clustering and a bundle factor of 4
DAX /Abstract Workflow
Number of Jobs by Type for 0.5 degree workflow |
|
|---|---|
Type of Job |
Number |
mProject |
15 |
mDiffFit |
29 |
mConcatFit |
1 |
mBgModel |
1 |
mBackground |
15 |
mImgTable |
1 |
mAdd |
1 |
mShrink |
1 |
mJPEG |
1 |
Total |
65 |
run0001 ( via CondorG without clustering )
PTC Numbers
Runtime Breakdown by job type
+-----------------------+-------------+--------+-------------+-------------+ | wf_label | tr_name | number | sum_seconds | avg_seconds | +-----------------------+-------------+--------+-------------+-------------+ | condorg_nocluster_0.5 | dirmanager | 1 | 0.10 | 0.10 | | condorg_nocluster_0.5 | mAdd | 1 | 11.60 | 11.60 | | condorg_nocluster_0.5 | mBackground | 15 | 110.99 | 7.40 | | condorg_nocluster_0.5 | mBgModel | 1 | 0.08 | 0.08 | | condorg_nocluster_0.5 | mConcatFit | 1 | 0.11 | 0.11 | | condorg_nocluster_0.5 | mDiffFit | 29 | 83.61 | 2.88 | | condorg_nocluster_0.5 | mImgtbl | 1 | 0.37 | 0.37 | | condorg_nocluster_0.5 | mJPEG | 1 | 1.01 | 1.01 | | condorg_nocluster_0.5 | mProjectPP | 15 | 162.38 | 10.83 | | condorg_nocluster_0.5 | mShrink | 1 | 2.26 | 2.26 | | condorg_nocluster_0.5 | rc-client | 3 | 0.72 | 0.24 | | condorg_nocluster_0.5 | transfer | 4 | 20.19 | 5.05 | +-----------------------+-------------+--------+-------------+-------------+
Number of failures by job type
Zero Failures
Number of successful jobs by job type
+-----------------------+-------------+--------+ | wf_label | tr_name | number | +-----------------------+-------------+--------+ | condorg_nocluster_0.5 | dirmanager | 1 | | condorg_nocluster_0.5 | mAdd | 1 | | condorg_nocluster_0.5 | mBackground | 15 | | condorg_nocluster_0.5 | mBgModel | 1 | | condorg_nocluster_0.5 | mConcatFit | 1 | | condorg_nocluster_0.5 | mDiffFit | 29 | | condorg_nocluster_0.5 | mImgtbl | 1 | | condorg_nocluster_0.5 | mJPEG | 1 | | condorg_nocluster_0.5 | mProjectPP | 15 | | condorg_nocluster_0.5 | mShrink | 1 | | condorg_nocluster_0.5 | rc-client | 3 | | condorg_nocluster_0.5 | transfer | 4 | +-----------------------+-------------+--------+
Netlogger Numbers
To be filled by Chris
Runtime Breakdown by job type
job name |
number |
total time |
average time |
|---|---|---|---|
mBgModel:3.0 |
1 |
0.085 |
0.085 |
pegasus::dirmanager |
1 |
0.104 |
0.104 |
mBackground:3.0 |
15 |
110.991 |
7.399 |
mProjectPP:3.0 |
15 |
162.377 |
10.825 |
pegasus::transfer |
4 |
20.189 |
5.048 |
pegasus::rc-client |
3 |
0.722 |
0.241 |
mDiffFit:3.0 |
29 |
83.606 |
2.883 |
mImgtbl:3.0 |
1 |
0.368 |
0.368 |
mAdd:3.0 |
1 |
11.605 |
11.605 |
mConcatFit:3.0 |
1 |
0.107 |
0.107 |
mShrink:3.0 |
1 |
2.265 |
2.265 |
mJPEG:3.0 |
1 |
1.009 |
1.009 |
Number of failures by job type
no failures
Number of successful jobs by job type
job name |
number |
|---|---|
mBgModel:3.0 |
1 |
pegasus::dirmanager |
1 |
mBackground:3.0 |
15 |
mProjectPP:3.0 |
15 |
pegasus::transfer |
4 |
pegasus::rc-client |
3 |
mDiffFit:3.0 |
29 |
mImgtbl:3.0 |
1 |
mAdd:3.0 |
1 |
mConcatFit:3.0 |
1 |
mShrink:3.0 |
1 |
mJPEG:3.0 |
1 |
run0002 ( via CondorG with clustering )
PTC Numbers
Runtime Breakdown by job type
+---------------------+-------------+--------+-------------+-------------+ | wf_label | tr_name | number | sum_seconds | avg_seconds | +---------------------+-------------+--------+-------------+-------------+ | condorg_cluster_0.5 | dirmanager | 1 | 0.07 | 0.07 | | condorg_cluster_0.5 | mAdd | 1 | 13.11 | 13.11 | | condorg_cluster_0.5 | mBackground | 15 | 75.20 | 5.01 | | condorg_cluster_0.5 | mBgModel | 1 | 0.11 | 0.11 | | condorg_cluster_0.5 | mConcatFit | 1 | 0.11 | 0.11 | | condorg_cluster_0.5 | mDiffFit | 29 | 70.65 | 2.44 | | condorg_cluster_0.5 | mImgtbl | 1 | 0.33 | 0.33 | | condorg_cluster_0.5 | mJPEG | 1 | 0.97 | 0.97 | | condorg_cluster_0.5 | mProjectPP | 15 | 97.37 | 6.49 | | condorg_cluster_0.5 | mShrink | 1 | 2.26 | 2.26 | | condorg_cluster_0.5 | rc-client | 3 | 0.73 | 0.24 | | condorg_cluster_0.5 | transfer | 4 | 19.00 | 4.75 | +---------------------+-------------+--------+-------------+-------------+
Number of failures by job type
Zero Failures
Number of successful jobs by job type
+---------------------+-------------+--------+ | wf_label | tr_name | number | +---------------------+-------------+--------+ | condorg_cluster_0.5 | dirmanager | 1 | | condorg_cluster_0.5 | mAdd | 1 | | condorg_cluster_0.5 | mBackground | 15 | | condorg_cluster_0.5 | mBgModel | 1 | | condorg_cluster_0.5 | mConcatFit | 1 | | condorg_cluster_0.5 | mDiffFit | 29 | | condorg_cluster_0.5 | mImgtbl | 1 | | condorg_cluster_0.5 | mJPEG | 1 | | condorg_cluster_0.5 | mProjectPP | 15 | | condorg_cluster_0.5 | mShrink | 1 | | condorg_cluster_0.5 | rc-client | 3 | | condorg_cluster_0.5 | transfer | 4 | +---------------------+-------------+--------+
Netlogger Numbers
To be filled by Chris
Runtime Breakdown by job type
name |
number |
total time |
average time |
|---|---|---|---|
mBgModel:3.0 |
1 |
0.112 |
0.112 |
mProjectPP:3.0 |
15 |
97.373 |
6.4915 |
mBackground:3.0 |
15 |
75.198 |
5.0132 |
pegasus::dirmanager |
1 |
0.072 |
0.072 |
pegasus::transfer |
4 |
19.003 |
4.751 |
pegasus::rc-client |
3 |
0.733 |
0.244 |
mDiffFit:3.0 |
29 |
70.647 |
2.436 |
mImgtbl:3.0 |
1 |
0.329 |
0.329 |
mAdd:3.0 |
1 |
13.107 |
13.107 |
mConcatFit:3.0 |
1 |
0.107 |
0.107 |
mShrink:3.0 |
1 |
2.258 |
2.258 |
mJPEG:3.0 |
1 |
0.97 |
0.97 |
Number of failures by job type
no failures
Number of successful jobs by job type
job name |
number |
|---|---|
mBgModel:3.0 |
1 |
pegasus::dirmanager |
1 |
mBackground:3.0 |
15 |
mProjectPP:3.0 |
15 |
pegasus::transfer |
4 |
pegasus::rc-client |
3 |
mDiffFit:3.0 |
29 |
mImgtbl:3.0 |
1 |
mAdd:3.0 |
1 |
mConcatFit:3.0 |
1 |
mShrink:3.0 |
1 |
mJPEG:3.0 |
1 |
run0003 ( via Condor Glidein without clustering )
PTC Numbers
Runtime Breakdown by job type
+-----------------------+-------------+--------+-------------+-------------+ | wf_label | tr_name | number | sum_seconds | avg_seconds | +-----------------------+-------------+--------+-------------+-------------+ | glidein_nocluster_0.5 | dirmanager | 1 | 0.14 | 0.14 | | glidein_nocluster_0.5 | mAdd | 1 | 11.43 | 11.43 | | glidein_nocluster_0.5 | mBackground | 15 | 34.63 | 2.31 | | glidein_nocluster_0.5 | mBgModel | 1 | 0.11 | 0.11 | | glidein_nocluster_0.5 | mConcatFit | 1 | 0.07 | 0.07 | | glidein_nocluster_0.5 | mDiffFit | 29 | 49.40 | 1.70 | | glidein_nocluster_0.5 | mImgtbl | 1 | 0.53 | 0.53 | | glidein_nocluster_0.5 | mJPEG | 1 | 1.01 | 1.01 | | glidein_nocluster_0.5 | mProjectPP | 15 | 72.61 | 4.84 | | glidein_nocluster_0.5 | mShrink | 1 | 3.58 | 3.58 | | glidein_nocluster_0.5 | rc-client | 9 | 2.19 | 0.24 | | glidein_nocluster_0.5 | transfer | 4 | 19.64 | 4.91 | +-----------------------+-------------+--------+-------------+-------------+
Number of failures by job type
+-----------------------+-----------+--------+ | wf_label | tr_name | number | +-----------------------+-----------+--------+ | glidein_nocluster_0.5 | rc-client | 9 | +-----------------------+-----------+--------+
Number of successful jobs by job type
+-----------------------+-------------+--------+ | wf_label | tr_name | number | +-----------------------+-------------+--------+ | glidein_nocluster_0.5 | dirmanager | 1 | | glidein_nocluster_0.5 | mAdd | 1 | | glidein_nocluster_0.5 | mBackground | 15 | | glidein_nocluster_0.5 | mBgModel | 1 | | glidein_nocluster_0.5 | mConcatFit | 1 | | glidein_nocluster_0.5 | mDiffFit | 29 | | glidein_nocluster_0.5 | mImgtbl | 1 | | glidein_nocluster_0.5 | mJPEG | 1 | | glidein_nocluster_0.5 | mProjectPP | 15 | | glidein_nocluster_0.5 | mShrink | 1 | | glidein_nocluster_0.5 | transfer | 4 | +-----------------------+-------------+--------+
Netlogger Numbers
To be filled by Chris
Runtime Breakdown by job type
job name |
number |
total time |
average time |
|---|---|---|---|
mAdd:3.0 |
1 |
11.427 |
11.427 |
mBackground:3.0 |
15 |
34.626 |
2.3084 |
mBgModel:3.0 |
1 |
0.11 |
0.11 |
mConcatFit:3.0 |
1 |
0.071 |
0.071 |
mDiffFit:3.0 |
29 |
49.4 |
1.703 |
mImgtbl:3.0 |
1 |
0.526 |
0.526 |
mJPEG:3.0 |
1 |
1.008 |
1.008 |
mProjectPP:3.0 |
15 |
72.61 |
4.840 |
mShrink:3.0 |
1 |
3.576 |
3.576 |
pegasus::dirmanager |
1 |
0.143 |
0.143 |
pegasus::rc-client |
9 |
2.186 |
0.243 |
pegasus::transfer |
4 |
19.64 |
4.91 |
Number of failures by job type
job |
number |
|---|---|
pegasus::rc-client |
9 |
Number of successful jobs by job type
job name |
number |
|---|---|
mAdd:3.0 |
1 |
mBackground:3.0 |
15 |
mBgModel:3.0 |
1 |
mConcatFit:3.0 |
1 |
mDiffFit:3.0 |
29 |
mImgtbl:3.0 |
1 |
mJPEG:3.0 |
1 |
mProjectPP:3.0 |
15 |
mShrink:3.0 |
1 |
pegasus::dirmanager |
1 |
pegasus::transfer |
4 |
run0004 ( via Condor glidein with clustering )
PTC Numbers
Runtime Breakdown by job type
+---------------------+-------------+--------+-------------+-------------+ | wf_label | tr_name | number | sum_seconds | avg_seconds | +---------------------+-------------+--------+-------------+-------------+ | glidein_cluster_0.5 | dirmanager | 1 | 0.12 | 0.12 | | glidein_cluster_0.5 | mAdd | 1 | 12.92 | 12.92 | | glidein_cluster_0.5 | mBackground | 15 | 60.55 | 4.04 | | glidein_cluster_0.5 | mBgModel | 1 | 0.08 | 0.08 | | glidein_cluster_0.5 | mConcatFit | 1 | 0.12 | 0.12 | | glidein_cluster_0.5 | mDiffFit | 29 | 47.72 | 1.65 | | glidein_cluster_0.5 | mImgtbl | 1 | 0.66 | 0.66 | | glidein_cluster_0.5 | mJPEG | 1 | 3.47 | 3.47 | | glidein_cluster_0.5 | mProjectPP | 15 | 72.22 | 4.81 | | glidein_cluster_0.5 | mShrink | 1 | 3.04 | 3.04 | | glidein_cluster_0.5 | rc-client | 9 | 2.20 | 0.24 | | glidein_cluster_0.5 | transfer | 4 | 18.18 | 4.54 | +---------------------+-------------+--------+-------------+-------------+
Number of failures by job type
+---------------------+-----------+--------+ | wf_label | tr_name | number | +---------------------+-----------+--------+ | glidein_cluster_0.5 | rc-client | 9 | +---------------------+-----------+--------+
Number of successful jobs by job type
+---------------------+-------------+--------+ | wf_label | tr_name | number | +---------------------+-------------+--------+ | glidein_cluster_0.5 | dirmanager | 1 | | glidein_cluster_0.5 | mAdd | 1 | | glidein_cluster_0.5 | mBackground | 15 | | glidein_cluster_0.5 | mBgModel | 1 | | glidein_cluster_0.5 | mConcatFit | 1 | | glidein_cluster_0.5 | mDiffFit | 29 | | glidein_cluster_0.5 | mImgtbl | 1 | | glidein_cluster_0.5 | mJPEG | 1 | | glidein_cluster_0.5 | mProjectPP | 15 | | glidein_cluster_0.5 | mShrink | 1 | | glidein_cluster_0.5 | transfer | 4 | +---------------------+-------------+--------+
Netlogger Numbers
To be filled by Chris
Runtime Breakdown by job type
job name |
number |
total time |
average time |
|---|---|---|---|
mAdd:3.0 |
1 |
12.919 |
12.919 |
mBackground:3.0 |
15 |
60.552 |
4.0368 |
mBgModel:3.0 |
1 |
0.077 |
0.077 |
mConcatFit:3.0 |
1 |
0.12 |
0.12 |
mDiffFit:3.0 |
29 |
47.716 |
1.645 |
mImgtbl:3.0 |
1 |
0.663 |
0.663 |
mJPEG:3.0 |
1 |
3.469 |
3.469 |
mProjectPP:3.0 |
15 |
72.22 |
4.814 |
mShrink:3.0 |
1 |
3.045 |
3.045 |
pegasus::dirmanager |
1 |
0.125 |
0.125 |
pegasus::transfer |
4 |
18.176 |
4.544 |
pegasus::rc-client |
9 |
2.202 |
0.245 |
Number of failures by job type
job |
number |
|---|---|
pegasus::rc-client |
9 |
Number of successful jobs by job type
job name |
number |
|---|---|
mAdd:3.0 |
1 |
mBackground:3.0 |
15 |
mBgModel:3.0 |
1 |
mConcatFit:3.0 |
1 |
mDiffFit:3.0 |
29 |
mImgtbl:3.0 |
1 |
mJPEG:3.0 |
1 |
mProjectPP:3.0 |
15 |
mShrink:3.0 |
1 |
pegasus::dirmanager |
1 |
pegasus::transfer |
4 |