• Created by Unknown User (gmehta), last modified on Oct 06, 2009

Netlogger Analysis of IHope Workflow

IHope Workflow

Gravitational Wave Metadata

Consists of 5 sub workflows of which 4 are executed via Pegasus

  • inspiral_hipe_eobinj.EOBINJ
  • inspiral_hipe_eobinj_cat2_veto.EOBINJ_CAT_2_VETO
  • inspiral_hipe_eobinj_cat3_veto.EOBINJ_CAT_3_VETO
  • inspiral_hipe_eobinj_cat4_veto.EOBINJ_CAT_4_VETO

Netlogger Numbers

  • Executed on site LIGO_UWM_NEMO

    inspiral_hipe_eobinj.EOBINJ

    Runtime Breakdown by job type

    +-----------------------------+--------+-------------+-----------+-------------+
| TRANSFORMATION              | number | sum_seconds | sum_hours | avg_seconds |
+-----------------------------+--------+-------------+-----------+-------------+
| ligo::lalapps_coire:1.0     |    402 |     1242.30 |      0.35 |        3.09 | 
| ligo::lalapps_inca:1.0      |     46 |       55.31 |      0.02 |        1.20 | 
| ligo::lalapps_inspinj:1.0   |      1 |        1.03 |      0.00 |        1.03 | 
| ligo::lalapps_inspiral:1.0  |   1621 |  2245439.14 |    623.73 |     1385.22 | 
| ligo::lalapps_sire:1.0      |   1296 |      723.78 |      0.20 |        0.56 | 
| ligo::lalapps_thinca:1.0    |    398 |     5413.69 |      1.50 |       13.60 | 
| ligo::lalapps_tmpltbank:1.0 |    736 |   399980.05 |    111.11 |      543.45 | 
| ligo::lalapps_trigbank:1.0  |    885 |     2253.89 |      0.63 |        2.55 | 
| pegasus::dirmanager         |      8 |        1.05 |      0.00 |        0.13 | 
| pegasus::transfer           |      9 |    24309.92 |      6.75 |     2701.10 | 
+-----------------------------+--------+-------------+-----------+-------------+

    Number of failures by job type

    No failures

    Number of successful jobs by job type

    +-----------------------------+---------+
| TRANSFORMATION              | success |
+-----------------------------+---------+
| ligo::lalapps_coire:1.0     |     402 | 
| ligo::lalapps_inca:1.0      |      46 | 
| ligo::lalapps_inspinj:1.0   |       1 | 
| ligo::lalapps_inspiral:1.0  |    1621 | 
| ligo::lalapps_sire:1.0      |    1296 | 
| ligo::lalapps_thinca:1.0    |     398 | 
| ligo::lalapps_tmpltbank:1.0 |     736 | 
| ligo::lalapps_trigbank:1.0  |     885 | 
| pegasus::dirmanager         |       8 | 
| pegasus::transfer           |       9 | 
+-----------------------------+---------+

Jobs Per Day Per Hour Per Workflow 

select day(from_unixtime(time)) as day, hour(from_unixtime(time)) as hour, count(event.id) as 'count'  from event
{panel}
           join attr on attr.e_id = event.id 
           join ident on attr.e_id=ident.e_id 
{panel}
where event.name = 'pegasus.invocation' and attr.name = 'host' and ident.name='workflow' and ident.value LIKE 'inspiral_hipe_eobinj.EOBINJ' 
group by hour, day ORDER BY day, hour;

+------+------+-------+
| day  | hour | count |
+------+------+-------+
|   23 |   12 |     1 | 
|   23 |   18 |   373 | 
|   23 |   19 |   420 | 
|   23 |   20 |   461 | 
|   23 |   21 |   414 | 
|   23 |   22 |   545 | 
|   23 |   23 |   399 | 
|   24 |    0 |   440 | 
|   24 |    1 |   413 | 
|   24 |    2 |   336 | 
|   24 |    3 |   415 | 
|   24 |    4 |   395 | 
|   24 |    5 |   395 | 
|   24 |    6 |   378 | 
|   24 |    7 |    17 | 
+------+------+-------+

Jobs Per Host Per Hour Per Workflow

select  attr.value  as host, day(from_unixtime(time)) as 'day', hour(from_unixtime(time)) as 'hour', count(event.id) as 'count'  from  event 
{panel}
           join attr on attr.e_id = event.id
           join ident on attr.e_id=ident.e_id
 where event.name = 'pegasus.invocation' and attr.name = 'host' and ident.name='workflow' and ident.value LIKE 'inspiral_hipe_eobinj.EOBINJ'
 group by host, day,hour  ORDER BY day, hour;
{panel}

inspiral_hipe_eobinj_cat2_veto.EOBINJ_CAT_2_VETO

Runtime Breakdown by job type

+--------------------------+--------+-------------+-----------+-------------+
| TRANSFORMATION           | number | sum_seconds | sum_hours | avg_seconds |
+--------------------------+--------+-------------+-----------+-------------+
| ligo::lalapps_coire:1.0  |    203 |      555.52 |      0.15 |        2.74 | 
| ligo::lalapps_sire:1.0   |    510 |       82.85 |      0.02 |        0.16 | 
| ligo::lalapps_thinca:1.0 |    199 |     1718.31 |      0.48 |        8.63 | 
| pegasus::dirmanager      |      3 |        0.24 |      0.00 |        0.08 | 
| pegasus::transfer        |      4 |      487.18 |      0.14 |      121.80 | 
+--------------------------+--------+-------------+-----------+-------------+

Number of failures by job type

No Failures

Number of successful jobs by job type

+--------------------------+---------+
| TRANSFORMATION           | success |
+--------------------------+---------+
| ligo::lalapps_coire:1.0  |     203 | 
| ligo::lalapps_sire:1.0   |     510 | 
| ligo::lalapps_thinca:1.0 |     199 | 
| pegasus::dirmanager      |       3 | 
| pegasus::transfer        |       4 | 
+--------------------------+---------+

Jobs Per Day Per Hour Per Workflow 

+------+------+-------+
| day  | hour | count |
+------+------+-------+
|   24 |    7 |   124 | 
|   24 |    8 |   340 | 
|   24 |    9 |   388 | 
|   24 |   10 |    67 | 
+------+------+-------+

Jobs Per Host Per Hour Per Workflow

inspiral_hipe_eobinj_cat3_veto.EOBINJ_CAT_3_VETO

Runtime Breakdown by job type

+--------------------------+--------+-------------+-----------+-------------+
| TRANSFORMATION           | number | sum_seconds | sum_hours | avg_seconds |
+--------------------------+--------+-------------+-----------+-------------+
| ligo::lalapps_coire:1.0  |    203 |      541.44 |      0.15 |        2.67 | 
| ligo::lalapps_sire:1.0   |    512 |       81.60 |      0.02 |        0.16 | 
| ligo::lalapps_thinca:1.0 |    199 |     1704.44 |      0.47 |        8.57 | 
| pegasus::dirmanager      |      3 |        0.25 |      0.00 |        0.08 | 
| pegasus::transfer        |      4 |      497.67 |      0.14 |      124.42 | 
+--------------------------+--------+-------------+-----------+-------------+

Number of failures by job type

No failures

Number of successful jobs by job type

+--------------------------+---------+
| TRANSFORMATION           | success |
+--------------------------+---------+
| ligo::lalapps_coire:1.0  |     203 | 
| ligo::lalapps_sire:1.0   |     512 | 
| ligo::lalapps_thinca:1.0 |     199 | 
| pegasus::dirmanager      |       3 | 
| pegasus::transfer        |       4 | 
+--------------------------+---------+

Jobs Per Day Per Hour Per Workflow 

+------+------+-------+
| day  | hour | count |
+------+------+-------+
|   24 |   10 |   144 | 
|   24 |   11 |   335 | 
|   24 |   12 |   373 | 
|   24 |   13 |    69 | 
+------+------+-------+

Jobs Per Host Per Hour Per Workflow

inspiral_hipe_eobinj_cat4_veto.EOBINJ_CAT_4_VETO

Runtime Breakdown by job type

+--------------------------+--------+-------------+-----------+-------------+
| TRANSFORMATION           | number | sum_seconds | sum_hours | avg_seconds |
+--------------------------+--------+-------------+-----------+-------------+
| ligo::lalapps_coire:1.0  |    203 |      536.77 |      0.15 |        2.64 | 
| ligo::lalapps_sire:1.0   |    510 |       81.21 |      0.02 |        0.16 | 
| ligo::lalapps_thinca:1.0 |    199 |     1691.11 |      0.47 |        8.50 | 
| pegasus::dirmanager      |      3 |        0.24 |      0.00 |        0.08 | 
| pegasus::transfer        |      4 |      488.96 |      0.14 |      122.24 | 
+--------------------------+--------+-------------+-----------+-------------+

Number of failures by job type

No Failures

Number of successful jobs by job type

+--------------------------+---------+
| TRANSFORMATION           | success |
+--------------------------+---------+
| ligo::lalapps_coire:1.0  |     203 | 
| ligo::lalapps_sire:1.0   |     510 | 
| ligo::lalapps_thinca:1.0 |     199 | 
| pegasus::dirmanager      |       3 | 
| pegasus::transfer        |       4 | 
+--------------------------+---------+

Jobs Per Day Per Hour Per Workflow 

+------+------+-------+
| day  | hour | count |
+------+------+-------+
|   24 |   13 |   159 | 
|   24 |   14 |   334 | 
|   24 |   15 |   380 | 
|   24 |   16 |    46 | 
+------+------+-------+

Jobs Per Host Per Hour Per Workflow

IHope Workflow ( All Four Sub Workflows executed via Pegasus)

Runtime Breakdown by job type

+-----------------------------+--------+-------------+-----------+-------------+
| TRANSFORMATION              | number | sum_seconds | sum_hours | avg_seconds |
+-----------------------------+--------+-------------+-----------+-------------+
| ligo::lalapps_coire:1.0     |   1011 |     2876.02 |      0.80 |        2.84 | 
| ligo::lalapps_inca:1.0      |     46 |       55.31 |      0.02 |        1.20 | 
| ligo::lalapps_inspinj:1.0   |      1 |        1.03 |      0.00 |        1.03 | 
| ligo::lalapps_inspiral:1.0  |   1621 |  2245439.14 |    623.73 |     1385.22 | 
| ligo::lalapps_sire:1.0      |   2828 |      969.44 |      0.27 |        0.34 | 
| ligo::lalapps_thinca:1.0    |    995 |    10527.56 |      2.92 |       10.58 | 
| ligo::lalapps_tmpltbank:1.0 |    736 |   399980.05 |    111.11 |      543.45 | 
| ligo::lalapps_trigbank:1.0  |    885 |     2253.89 |      0.63 |        2.55 | 
| pegasus::dirmanager         |     17 |        1.77 |      0.00 |        0.10 | 
| pegasus::transfer           |     21 |    25783.74 |      7.16 |     1227.80 | 
+-----------------------------+--------+-------------+-----------+-------------+

Number of failures by job type

No Failures

Number of successful jobs by job type

+-----------------------------+---------+
| TRANSFORMATION              | success |
+-----------------------------+---------+
| ligo::lalapps_coire:1.0     |    1011 | 
| ligo::lalapps_inca:1.0      |      46 | 
| ligo::lalapps_inspinj:1.0   |       1 | 
| ligo::lalapps_inspiral:1.0  |    1621 | 
| ligo::lalapps_sire:1.0      |    2828 | 
| ligo::lalapps_thinca:1.0    |     995 | 
| ligo::lalapps_tmpltbank:1.0 |     736 | 
| ligo::lalapps_trigbank:1.0  |     885 | 
| pegasus::dirmanager         |      17 | 
| pegasus::transfer           |      21 | 
+-----------------------------+---------+

Jobs Per Day Per Hour Per Workflow 

+------+------+-------+
| day  | hour | count |
+------+------+-------+
|   23 |   12 |     1 | 
|   23 |   18 |   373 | 
|   23 |   19 |   420 | 
|   23 |   20 |   461 | 
|   23 |   21 |   414 | 
|   23 |   22 |   545 | 
|   23 |   23 |   399 | 
|   24 |    0 |   440 | 
|   24 |    1 |   413 | 
|   24 |    2 |   336 | 
|   24 |    3 |   415 | 
|   24 |    4 |   395 | 
|   24 |    5 |   395 | 
|   24 |    6 |   378 | 
|   24 |    7 |   141 | 
|   24 |    8 |   340 | 
|   24 |    9 |   388 | 
|   24 |   10 |   211 | 
|   24 |   11 |   335 | 
|   24 |   12 |   373 | 
|   24 |   13 |   228 | 
|   24 |   14 |   334 | 
|   24 |   15 |   380 | 
|   24 |   16 |    46 | 
+------+------+-------+

Jobs Per Host Per Hour Per Workflow

Jobs Per Host

SELECT value as host, count(event.id) as count FROM event
{panel}
              JOIN attr on attr.e_id = event.id 
{panel}
WHERE event.name = 'pegasus.invocation' and attr.name = 'host' 
GROUP BY host ORDER BY count; 

+----------------------------------+-------+
| host                             | count |
+----------------------------------+-------+
| nemo-slave0302.nemo.phys.uwm.edu |  1081 | 
| nemo-slave0642.nemo.phys.uwm.edu |   975 | 
| nemo-slave0473.nemo.phys.uwm.edu |   778 | 
| nemo-slave0565.nemo.phys.uwm.edu |   601 | 
| nemo-slave0705.nemo.phys.uwm.edu |   504 | 
| nemo-slave0349.nemo.phys.uwm.edu |   410 | 
| nemo-slave0580.nemo.phys.uwm.edu |   410 | 
| nemo-slave0431.nemo.phys.uwm.edu |   272 | 
| nemo-slave0439.nemo.phys.uwm.edu |   223 | 
| nemo-slave0393.nemo.phys.uwm.edu |   203 | 
| nemo-slave0257.nemo.phys.uwm.edu |   144 | 
| nemo-slave0080.nemo.phys.uwm.edu |   143 | 
| nemo-slave0388.nemo.phys.uwm.edu |   131 | 
| nemo-slave0650.nemo.phys.uwm.edu |   111 | 
| nemo-slave0258.nemo.phys.uwm.edu |   109 | 
| nemo-slave0321.nemo.phys.uwm.edu |   107 | 
| nemo-slave0402.nemo.phys.uwm.edu |    81 | 
| nemo-slave0377.nemo.phys.uwm.edu |    66 | 
| nemo-slave0299.nemo.phys.uwm.edu |    61 | 
| nemo-slave0318.nemo.phys.uwm.edu |    48 | 
| nemo-slave0152.nemo.phys.uwm.edu |    47 | 
| nemo-slave0669.nemo.phys.uwm.edu |    47 | 
| nemo-slave0530.nemo.phys.uwm.edu |    46 | 
| nemo-slave0646.nemo.phys.uwm.edu |    46 | 
| nemo-slave0071.nemo.phys.uwm.edu |    46 | 
| nemo-slave0699.nemo.phys.uwm.edu |    43 | 
| nemo-slave0054.nemo.phys.uwm.edu |    42 | 
| nemo-slave0746.nemo.phys.uwm.edu |    42 | 
| nemo-slave0260.nemo.phys.uwm.edu |    39 | 
| nemo-slave0498.nemo.phys.uwm.edu |    38 | 
| osg-nemo-ce.phys.uwm.edu         |    38 | 
| nemo-slave0158.nemo.phys.uwm.edu |    37 | 
| nemo-slave0111.nemo.phys.uwm.edu |    37 | 
| nemo-slave0009.nemo.phys.uwm.edu |    37 | 
| nemo-slave0194.nemo.phys.uwm.edu |    34 | 
| nemo-slave0162.nemo.phys.uwm.edu |    34 | 
| nemo-slave0489.nemo.phys.uwm.edu |    34 | 
| nemo-slave0634.nemo.phys.uwm.edu |    33 | 
| nemo-slave0490.nemo.phys.uwm.edu |    31 | 
| nemo-slave0697.nemo.phys.uwm.edu |    31 | 
| nemo-slave0118.nemo.phys.uwm.edu |    29 | 
| nemo-slave0232.nemo.phys.uwm.edu |    29 | 
| nemo-slave0400.nemo.phys.uwm.edu |    28 | 
| nemo-slave0376.nemo.phys.uwm.edu |    26 | 
| nemo-slave0139.nemo.phys.uwm.edu |    26 | 
| nemo-slave0124.nemo.phys.uwm.edu |    26 | 
| nemo-slave0467.nemo.phys.uwm.edu |    26 | 
| nemo-slave0123.nemo.phys.uwm.edu |    23 | 
| nemo-slave0330.nemo.phys.uwm.edu |    23 | 
| nemo-slave0613.nemo.phys.uwm.edu |    23 | 
| nemo-slave0170.nemo.phys.uwm.edu |    21 | 
| nemo-slave0316.nemo.phys.uwm.edu |    20 | 
| nemo-slave0745.nemo.phys.uwm.edu |    20 | 
| nemo-slave0212.nemo.phys.uwm.edu |    20 | 
| nemo-slave0197.nemo.phys.uwm.edu |    20 | 
| nemo-slave0492.nemo.phys.uwm.edu |    19 | 
| nemo-slave0183.nemo.phys.uwm.edu |    19 | 
| nemo-slave0517.nemo.phys.uwm.edu |    19 | 
| nemo-slave0246.nemo.phys.uwm.edu |    18 | 
| nemo-slave0497.nemo.phys.uwm.edu |    18 | 
| nemo-slave0398.nemo.phys.uwm.edu |    17 | 
| nemo-slave0652.nemo.phys.uwm.edu |    17 | 
| nemo-slave0714.nemo.phys.uwm.edu |    17 | 
| nemo-slave0438.nemo.phys.uwm.edu |    17 | 
| nemo-slave0196.nemo.phys.uwm.edu |    16 | 
| nemo-slave0567.nemo.phys.uwm.edu |    16 | 
| nemo-slave0675.nemo.phys.uwm.edu |    15 | 
| nemo-slave0027.nemo.phys.uwm.edu |    14 | 
| nemo-slave0559.nemo.phys.uwm.edu |    14 | 
| nemo-slave0271.nemo.phys.uwm.edu |    14 | 
| nemo-slave0156.nemo.phys.uwm.edu |    13 | 
| nemo-slave0608.nemo.phys.uwm.edu |    13 | 
| nemo-slave0470.nemo.phys.uwm.edu |    13 | 
| nemo-slave0100.nemo.phys.uwm.edu |    12 | 
| nemo-slave0233.nemo.phys.uwm.edu |    12 | 
| nemo-slave0686.nemo.phys.uwm.edu |    12 | 
| nemo-slave0481.nemo.phys.uwm.edu |    12 | 
| nemo-slave0737.nemo.phys.uwm.edu |    12 | 
| nemo-slave0521.nemo.phys.uwm.edu |    10 | 
| nemo-slave0735.nemo.phys.uwm.edu |     9 | 
| nemo-slave0765.nemo.phys.uwm.edu |     9 | 
| nemo-slave0453.nemo.phys.uwm.edu |     9 | 
| nemo-slave0256.nemo.phys.uwm.edu |     9 | 
| nemo-slave0618.nemo.phys.uwm.edu |     9 | 
| nemo-slave0031.nemo.phys.uwm.edu |     9 | 
| nemo-slave0589.nemo.phys.uwm.edu |     9 | 
| nemo-slave0736.nemo.phys.uwm.edu |     8 | 
| nemo-slave0261.nemo.phys.uwm.edu |     8 | 
| nemo-slave0060.nemo.phys.uwm.edu |     8 | 
| nemo-slave0277.nemo.phys.uwm.edu |     7 | 
| nemo-slave0525.nemo.phys.uwm.edu |     7 | 
| nemo-slave0709.nemo.phys.uwm.edu |     6 | 
| nemo-slave0469.nemo.phys.uwm.edu |     6 | 
| nemo-slave0097.nemo.phys.uwm.edu |     6 | 
| nemo-slave0457.nemo.phys.uwm.edu |     6 | 
| nemo-slave0577.nemo.phys.uwm.edu |     5 | 
| nemo-slave0772.nemo.phys.uwm.edu |     5 | 
| nemo-slave0026.nemo.phys.uwm.edu |     5 | 
| nemo-slave0379.nemo.phys.uwm.edu |     5 | 
| nemo-slave0722.nemo.phys.uwm.edu |     4 | 
| nemo-slave0115.nemo.phys.uwm.edu |     4 | 
| nemo-slave0053.nemo.phys.uwm.edu |     4 | 
| nemo-slave0726.nemo.phys.uwm.edu |     4 | 
| nemo-slave0215.nemo.phys.uwm.edu |     3 | 
| nemo-slave0224.nemo.phys.uwm.edu |     3 | 
| nemo-slave0242.nemo.phys.uwm.edu |     3 | 
| nemo-slave0614.nemo.phys.uwm.edu |     3 | 
| nemo-slave0503.nemo.phys.uwm.edu |     3 | 
| nemo-slave0119.nemo.phys.uwm.edu |     3 | 
| nemo-slave0290.nemo.phys.uwm.edu |     3 | 
| nemo-slave0223.nemo.phys.uwm.edu |     2 | 
| nemo-slave0405.nemo.phys.uwm.edu |     2 | 
| nemo-slave0708.nemo.phys.uwm.edu |     2 | 
| nemo-slave0731.nemo.phys.uwm.edu |     2 | 
| nemo-slave0253.nemo.phys.uwm.edu |     2 | 
| nemo-slave0526.nemo.phys.uwm.edu |     2 | 
| nemo-slave0281.nemo.phys.uwm.edu |     2 | 
| nemo-slave0685.nemo.phys.uwm.edu |     2 | 
| nemo-slave0040.nemo.phys.uwm.edu |     2 | 
| nemo-slave0028.nemo.phys.uwm.edu |     1 | 
| nemo-slave0313.nemo.phys.uwm.edu |     1 | 
+----------------------------------+-------+
121 rows in set (0.15 sec)

Visualization of runs over time

X axis - time in seconds

Y axis - number of jobs

inspiral_hipe_eobinj_cat2_veto.EOBINJ_CAT_2_VETO

http://www.isi.edu/~vahi/work/netlogger_ligo/inspiral_hipe_eobinj_cat2_veto.EOBINJ_CAT_2_VETO.jpg

inspiral_hipe_eobinj_cat3_veto.EOBINJ_CAT_3_VETO

http://www.isi.edu/~vahi/work/netlogger_ligo/inspiral_hipe_eobinj_cat3_veto.EOBINJ_CAT_3_VETO.jpg

inspiral_hipe_eobinj_cat4_veto.EOBINJ_CAT_4_VETO

http://www.isi.edu/~vahi/work/netlogger_ligo/inspiral_hipe_eobinj_cat4_veto.EOBINJ_CAT_4_VETO.jpg

inspiral_hipe_eobinj.EOBINJ

http://www.isi.edu/~vahi/work/netlogger_ligo/inspiral_hipe_eobinj.EOBINJ.png

Visualization of jobs over nodes in the cluster

EPS
JPG
PLS
DAT

Visualization of Total Jobs executed Over Time

EPS
JPG
PLS
DAT

  • No labels