Introduction
Collaboration between Pegasus and the Center for Enabling Distributed Petascale Science (CEDPS) troubleshooting project, whose software is almost all part of NetLogger. For the converse, see the Pegasus page on the CEDPS wiki.
http://www.cedps.net/index.php/Pegasus/Kickstart_Log_Processing
PEGASUS SVN
https://smarty.isi.edu/svn/repo1/pegasus/trunk
the new parser is called contrib/showlog/gensim
SCEC RUNS
START and END TIMES OF RUNS
for site in USC SABD LADT WNGC FFI CCP SBSM SMCA LBP; do
{panel}
echo "Site $site";
mysql -u dbuser -ppwd -h isis.isi.edu cybershk_july_2008_2 -e
"select MIN(from_unixtime(time)) as START, MAX(from_unixtime(time)) as END from event
join ident on event.id=ident.e_id WHERE ident.value LIKE 'CyberShake_${site}%'";
echo "";echo""; done;
{panel}
Site USC +---------------------+---------------------+ | START | END | +---------------------+---------------------+ | 2008-07-07 16:59:46 | 2008-07-08 10:17:51 | +---------------------+---------------------+ Site SABD +---------------------+---------------------+ | START | END | +---------------------+---------------------+ | 2008-07-13 08:24:20 | 2008-07-14 00:53:58 | +---------------------+---------------------+ Site LADT +---------------------+---------------------+ | START | END | +---------------------+---------------------+ | 2008-07-12 13:08:23 | 2008-07-13 12:41:09 | +---------------------+---------------------+ Site WNGC +---------------------+---------------------+ | START | END | +---------------------+---------------------+ | 2008-07-11 18:52:22 | 2008-07-13 16:56:21 | +---------------------+---------------------+ Site FFI +---------------------+---------------------+ | START | END | +---------------------+---------------------+ | 2008-07-17 13:02:01 | 2008-07-18 10:29:12 | +---------------------+---------------------+ Site CCP +---------------------+---------------------+ | START | END | +---------------------+---------------------+ | 2008-07-16 21:17:59 | 2008-07-17 11:59:24 | +---------------------+---------------------+ Site SBSM +---------------------+---------------------+ | START | END | +---------------------+---------------------+ | 2008-07-16 07:47:14 | 2008-07-17 08:45:11 | +---------------------+---------------------+ Site SMCA +---------------------+---------------------+ | START | END | +---------------------+---------------------+ | 2008-07-15 13:06:01 | 2008-07-16 02:15:15 | +---------------------+---------------------+ Site LBP +---------------------+---------------------+ | START | END | +---------------------+---------------------+ | 2008-07-14 10:34:11 | 2008-07-15 13:02:48 | +---------------------+---------------------+
Complete RUNS July 2008
( With kickstart outputs /jobstate.log )
USC_PP
Netlogger Statistics
TRANSFORMATION | number | sum_seconds | sum_hours | avg_seconds | failures |
|---|---|---|---|---|---|
pegasus::dirmanager | 40 | 24.75 | 0.01 | 0.62 | 0 |
pegasus::rc-client | 80 | 207.61 | 0.06 | 2.6 | 0 |
pegasus::transfer | 384 | 66492.13 | 18.47 | 173.16 | 264 |
scec::extract_sgt:1.0 | 7000 | 1082070.75 | 300.58 | 154.58 | 0 |
scec::PeakValCalc_Okaya:1.0 | 417886 | 630647.75 | 175.18 | 1.51 | 0 |
scec::seismogram_synthesis:1.0 | 417886 | 26980412.48 | 7494.56 | 64.56 | 0 |
scec::ZipPeakSA:1.0 | 40 | 55347.38 | 15.37 | 1383.68 | 0 |
scec::ZipSeismograms:1.0 | 40 | 57728.22 | 16.04 | 1443.21 | 0 |
TOTALS | 843356 | 28872931.07 | 8020.27 | 264 |
TRANSFORMATION failures pegasus::transfer 264 Time to load to Gaurang Mac 2 hours 34 minutes
Gensim Statistics
AVERAGES | ||||||
|---|---|---|---|---|---|---|
Job Type | Job Count | Condor Delay | Dagman Delay | Seqexec Delay | Postscript Time | Internal Time |
Stage_in | 0 | NO ENTRIES | ||||
extract_sgt | 0 | NO ENTRIES | ||||
merge_scec-extract_sgt | 2348 | 198.27 | 173.02 | 19.88 | 5.18 | 480.73 |
merge_scec-seismogram_synthesis | 10464 | 515.48 | 504.24 | 771.68 | 5.02 | 3350.08 |
merge_scec-PeakValCalc_Okaya | 2107 | 541.6 | 94.79 | 1610.77 | 5.01 | 1910.08 |
Zip | 80 | 321.93 | 5.12 | 0 | 5.01 | 1413.45 |
stage_out | 0 | NO ENTRIES | ||||
other jobs | 280 | 59.37 | 2406.35 | 0 | 4.31 | 208.01 |
TOTALS | ||||||
|---|---|---|---|---|---|---|
Job Type | Job Count | Condor Delay | Dagman Delay | Seqexec Delay | Postscript Time | Internal Time |
stage_in | 0 | 0 | 0 | 0 | 0 | 0 |
extract_sgt | 0 | 0 | 0 | 0 | 0 | 0 |
merge_scec-extract_sgt | 2348 | 465544 | 406248 | 46674.47 | 12168 | 1128745.22 |
merge_scec-seismogram_synthesis | 10464 | 5394026 | 5276328 | 8074868.93 | 52565 | 35055281.69 |
merge_scec-PeakValCalc_Okaya | 2107 | 1141157 | 199729 | 3393894.22 | 10563 | 4024542.21 |
Zip | 80 | 25754 | 410 | 0 | 401 | 113075.61 |
stage_out | 0 | 0 | 0 | 0 | 0 | 0 |
other jobs | 280 | 16623 | 673779 | 0 | 1207 | 58242.78 |
Total (sec): | 15279 | 7043104 | 6556494 | 11515437.62 | 76904 | 40379887.5 |
Total (hr): | 15279 | 1956.42 | 1821.25 | 3198.73 | 21.36 | 11216.64 |
Percent: | 10.74% | 10.00% | 17.56% | 0.12% | 61.58% |