Skip to end of metadata
Go to start of metadata

Overview

This page describes how to run the Montage workflow. Please note that there are many ways of setting up these workflows, including using different Pegasus data staging approaches. A good place to get started is looking at the Montage examples shipped with Pegasus (under /usr/share/pegasus/examples/ if you installed from RPMs/DEBs).

Source Code

The current version of Montage which has been verified to work as a workflow is a lightly patched version. Pick up the latest from http://pegasus.isi.edu/montage/

Generating Montage DAX

The invocation is

mDAG exp channel object widgth height cdelta workdir wurl inputurl

where

  1. exp is 2mass
  2. channel is j, h, k
  3. object is like M17, M42 or one of the clusters.
  4. width and height are 1 and 1 for a 1 degree
  5. cdelta is always 0.000278
  6. wurl is the gridftp accessible url for the workdir to pick up certain header files
  7. inputurl is the accessible url for the input directory where the fits files (input data for the workflow got by ArchiveExec) are placed

The mDAG will create a cache.list file in the workdir, that can be used as a SimpleFile Replica Catalog to use with Pegasus

Example usage for newer versions of Montage ( 3.3 or higher )

mkdir 0.5
cd 0.5/
mkdir input workdir
mDAG 2mass j M17 0.5 0.5 0.000278 workdir "gsiftp://sukhna.isi.edu`pwd`" "gsiftp://sukhna.isi.edu`pwd`/input"

Creating the replica catalog

If you want the workflow to fetch the input data directly from the IPAC servers, you can create a replace catalog which has two major components: metadata files will come from your submit host, and the input image data will come from IPAC. To create such a replica catalog, run:

cat cache.list | grep -v ".fits " >rc
perl -p -i -e 's/ipac_cluster/local/' rc
cat url.list | sed 's/\.gz http/ http/' >>rc

Fetching the input fit files for the DAX

If you prefer to pre-fetch the data and put it on your submit host, the mArchiveExec command can be used to download the data. The images.tbl file contain the url's for the fit files that needs to be retrieved.

cp workdir/images.tbl input/
cd input/
mArchiveExec images.tbl

 

DAX Parsing and Planning Times

Machine Setup

  • Experiments were run on corbusier.isi.edu
    • Processor 2 x 2.0 GHz Quad-Core Intel Xeon
    • Memory 2 GB 800 MHz DDR2 FB-DIMM
    • OS Mac OSX Version 10.5.6
  • JAVA Heap Settings
    • Minimum 512 MB
    • Maximum 1024 MB

General DAX Metadata

  • Band j
  • Survey M17
  • exp 2mass

Pegasus Configuration

  • Version - 2.3.0cvs-x86_64_macos_10.5-20090130202332Z
  • Clustering - Off
  • Cleanup - Off
  • Reduction - Off
  • Replica Catalog - Cache File containing only input entries for the DAX in question
  • Transfer Mode - Bundle
  • Directory Creation - Tentacles
  • Site Selector - Random . Only 1 site passed for planning

Parsing and Planning Times

0.5 degree

# Montage 0.5 degree dax
# Size of dax           - 85K
# Number of Jobs in dax -       65
# All times in seconds
#         DAX Parsing Time   DAX Planning Time
run0001   .197                0.828
run0002   .199                0.856
run0003   .194                0.819
run0004   .198                0.892
run0005   .199                0.85
------------------------------------------------
Average   .197                .849

1.0 degree

# Montage 1.0 degree dax
# Size of dax           - 275K
# Number of Jobs in dax -      203
# All times in seconds
#         DAX Parsing Time   DAX Planning Time
run0001   .260                1.363
run0002   .262                1.231
run0003   .260                1.162
run0004   .256                1.088
run0005   .258                1.183
------------------------------------------------
Average   .259                1.205

2.0 degree

# Montage 2.0 degree dax
# Size of dax           - 926K
# Number of Jobs in dax -      675
# All times in seconds
#         DAX Parsing Time   DAX Planning Time
run0001   .400                1.891
run0002   .407                1.942
run0003   .381                1.887
run0004   .382                1.902
run0005   .397                1.907
------------------------------------------------
Average   .393                1.905

4.0 degree

# Montage 4.0 degree dax
# Size of dax           - 3.9M
# Number of Jobs in dax -     2860
# All times in seconds
#         DAX Parsing Time   DAX Planning Time
run0001   .955                7.675
run0002   .999                7.696
run0003   .955                7.636
run0004   .949                7.656
run0005   .963                7.563
------------------------------------------------
Average   .964                7.645

6.0 degree

# Montage 6.0 degree dax
# Size of dax           - 8.3M
# Number of Jobs in dax -     6059
# All times in seconds
#         DAX Parsing Time   DAX Planning Time
run0001   2.211                23.98
run0002   2.131                25.009
run0003   2.112                22.543
run0004   2.081                23.683
run0005   2.155                23.679
------------------------------------------------
Average   2.138                23.778

8.0 degree

# Montage 8.0 degree dax
# Size of dax           - 14M
# Number of Jobs in dax -    10429
# All times in seconds
#         DAX Parsing Time   DAX Planning Time
run0001   4.513                57.887
run0002   4.962                59.608
run0003   4.995                59.499
run0004   5.985                67.08
run0005   5.065                60.205
------------------------------------------------
Average   5.104                60.855

10.0 degree

# Montage 10.0 degree dax
# Size of dax           - 23M
# Number of Jobs in dax -    16879
# All times in seconds
#         DAX Parsing Time   DAX Planning Time
run0001   9.320                145.086
run0002   10.587                146.419
run0003   10.553                152.841
run0004   10.490                154.927
run0005   10.093                152.896
------------------------------------------------
Average   10.208                150.433

15.0 degree

# Montage 15.0 degree dax
# Size of dax           - 50M
# Number of Jobs in dax -    36597
# All times in seconds
#         DAX Parsing Time   DAX Planning Time
run0001   40.635                660.318
run0002   46.905                690.528
run0003   45.973                690.294
run0004   43.812                674.213
run0005   45.651                689.142
------------------------------------------------
Average   44.595                680.899

20.0 degree

# Montage 20.0 degree dax
# Size of dax           - 88M
# Number of Jobs in dax -    64220
# All times in seconds
#         DAX Parsing Time   DAX Planning Time
run0001   144.278                2134.931
run0002   162.236                2227.56
run0003   153.982                2225.636
run0004   155.489                2216.377
run0005   174.273                2316.763
------------------------------------------------
Average   158.051                2224.253

Summary

Summary

Degree

Size (MB)

Jobs

Avg DAX Parsing Time (seconds)

Avg DAX Planning Time (seconds)

0.5

0.083

65

0.197

0.849

1

0.268

203

0.259

1.205

2

0.904

675

0.393

1.905

4

3.9

2860

0.964

7.645

6

8.3

6059

2.138

23.778

8

14

10429

5.104

60.855

10

23

16879

10.208

150.433

15

50

36597

44.595

680.899

20

88

64220

158.051

2224.253

  • No labels