Child pages
  • Grid Setup

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.

... provides a very nice description of the firewalls and ports that need to be opened. It lists incoming and outgoing connection details. Note: You can specify PORT ranges for incoming connections (at least a hundered) plus the list of standard ports required for grid services. I would say to start we are setting up a small grid.

Once we have Globus running and tested, we should have a list of hosts on the TANGRAM Grid, with IP address and DNS, to allow Firewalls to be used effectively. We plan to run some monitoring software providing live status of the TANGRAM Grid.

As a group we could agree on a TCP PORTRANGE for Globus Connections.

Network Firewall Ports

These ports are to be opened bi-directional

  1. GRIDFTP 2811
  2. GATEKEEPER 2119
  3. GSISSH 40022
  4. GLOBUS TCP PORT RANGE = 40000,41000 or more if possible.
  5. Allegro Graph Server port = 4567
  6. MYSQL Server port = 3306
  7. HTTP ports (outgoing to connect to various services) 80, 8080, 8443, 443 etc
  8. GANGLIA ports 8655, 8649
  9. DC port ?
  10. PC port ?


  • Modify the gmond.conf file for your setup.
    • Edit the following section and put values for your cluster.
    • 2.x versions the following entries need to be changed name, owner, url, mcast, setuid.

      name "Viz"
      owner "ISI / CGT"
      url ""
      mcast_if eth0
      setuid ganglia

      • 3.x versions do following

        cluster {
        name = "Windward"
        owner = "ISI / CGT"
        latlong = "N30.0 W122.23"
        url = ""

  • edit the udp_send_channel section and change the mcast_join hostname/ip to be the hostname where your gmetad daemon is running.

    udp_send_channel {
    mcast_join =
    port = 8649
    ttl = 1


  • Edit the /etc/gmetad.conf file
    • 2.x the following entries need to be changed

      data_source "<Clustername>" localhost
      gridname "<your organization>"
      authority ""
      trusted_hosts <yourhostname or ip>
      setuid_username "ganglia"


Here is an example of what is displayed

Code Block

root@ttwo:/local/software/grid/condor/src/condor-7.1.3# condor_status

Name               OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime LINUX      INTEL  Unclaimed Idle     0.000  1013  0+00:05:04 LINUX      INTEL  Unclaimed Idle     0.000  1013  0+00:05:05

                     Total Owner Claimed Unclaimed Matched Preempting Backfill

         INTEL/LINUX     2     0       0         2       0          0        0

               Total     2     0       0         2       0          0        0

root@ttwo:/local/software/grid/condor/src/condor-7.1.3# condor_q

-- Submitter: : <> :


  • To start with
    • Write a file called globus-gatekeeper in /etc/xinetd.d directory
    • Replace <$TANGRAM_ROOT_DIR> with the value of the environment variable that you set
Code Block

service globus-gatekeeper
      socket_type  = stream
      protocol     = tcp
      wait         = no
      user         = root
      server       = <$TANGRAM_ROOT_DIR>/globus/default/sbin/globus-gatekeeper
      server_args  = -conf <$TANGRAM_ROOT_DIR>/globus/default/etc/globus-gatekeeper.conf
      disable      = no
      env          = LD_LIBRARY_PATH=<$TANGRAM_ROOT_DIR>/globus/default/lib
      env         += GLOBUS_LOCATION=<$TANGRAM_ROOT_DIR>/globus/default
     env         += GLOBUS_TCP_PORT_RANGE=40000,41000
    • Write a file called gridftp in the same directory
    • Replace <$TANGRAM_ROOT_DIR> with the value of the environment variable that you set
Code Block

service gridftp
            instances               = 100
            socket_type             = stream
            wait                    = no
            user                    = root
            server                  = <$TANGRAM_ROOT_DIR>/globus/default/sbin/globus-gridftp-server
            server_args             = -i -d info -l <~UWC_TOKEN_START~1254808297989~UWC_TOKEN_END~TANGRAM_ROOT_DIR>/globus/default/var/gridftp.log
            log_on_success         += DURATION USERID
            log_on_failure         += USERID
            nice                    = 10
            disable                 = no
            env                    += GLOBUS_LOCATION=<$TANGRAM_ROOT_DIR>/globus/default
            env                    += PATH=<~UWC_TOKEN_START~1254808297991~UWC_TOKEN_END~TANGRAM_ROOT_DIR>/globus/default/bin:<~UWC_TOKEN_START~1254808297992~UWC_TOKEN_END~TANGRAM_ROOT_DIR>/globus/default/sbin
            env                    += LD_LIBRARY_PATH=<~UWC_TOKEN_START~1254808297993~UWC_TOKEN_END~TANGRAM_ROOT_DIR>/globus/default/lib
           env         += GLOBUS_TCP_PORT_RANGE=40000,41000


  • source $GLOBUS_LOCATION/etc/
  • Run the command grid-cert-request -ca 0683a0c5 -host "fullhostname-including-domain". e.g.

    grid-cert-request -ca 0683a0c5 -host ""


  • source $GLOBUS_LOCATION/etc/
  • Run the command grid-cert-request -ca 0683a0c5 -cn "Your Full Name + 6 digitpin" . It will prompt you to specify a password. Put your password... e.g.

    grid-cert-request -ca 0683a0c5 -cn "Gaurang Mehta 123456"



Code Block

sukhna 32% grid-proxy-init
Your identity: /DC=org/DC=doegrids/OU=People/CN=Karan Vahi 476301
Enter GRID pass phrase for this identity:
Creating proxy ................................. Done
Your proxy is valid until: Sat Oct 25 03:03:12 2008

Testing the grid ftp server

sukhna 33% telnet 2811
Connected to
Escape character is '^]'.
220 GridFTP Server 2.8 (gcc32, 1217607445-63) [Globus Toolkit 4.0.8] ready.

Testing the jobmanager

sukhna 34%
sukhna 34% globusrun -a -r

GRAM Authentication test successful
sukhna 35%

sukhna 41% setenv GLOBUS_TCP_PORT_RANGE 40000,41000
sukhna 42% globus-job-run /bin/date
Fri Oct 24 18:09:43 EDT 2008

sukhna 43% globus-job-run /bin/date
Fri Oct 24 18:11:05 EDT 2008

 gsissh -p 40022
Warning: No xauth data; using fake authentication data for X11 forwarding.
Linux ttwo 2.6.24-etchnhalf.1-686 #1 SMP Mon Jul 21 11:17:43 UTC 2008 i686

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
/usr/bin/X11/xauth:  creating new authority file /home/sr/.Xauthority
sr@ttwo:~~UWC_TOKEN_START~1254808298002~UWC_TOKEN_END~ exit


NOTE: you will issue the commands from your machine containing the VDT installation.
(the following is for BASH shells, others CSH/TCSH should change these accordingly):

Code Block

~UWC_TOKEN_START~1254808298003~UWC_TOKEN_END~ export GLOBUS_LOCATION=<path_to_globus>

You need to have the credentials in the right place, and initialized

Code Block

$ mkdir ~/.globus

$ cp usercert.pem ~/.globus
$ cp userkey.pem ~/.globus
$ grid-proxy-init

You are ready to issued grid commands

Test the ISI Grid


  • Check if you can authenticate
    Code Block
    ~UWC_TOKEN_START~1254808298010~UWC_TOKEN_END~ globusrun -a -r
  • Check if you can run a job
    Code Block
    $ globus-job-run /bin/date
  • Check if you can transfer a file
    Code Block
    ~UWC_TOKEN_START~1254808298012~UWC_TOKEN_END~ globus-url-copy -dbg -vb file:///tmp/sometemp file gsi


Most of the time you will get the ill-famed

Code Block

GRAM Job submission failed because the job manager failed to open stderr (error code 74)

This is caused by the fact that you are firewalled, and you can address these in the following way: one trick you can use is to specify that your machine expects replies
on a specific range of ports:

Code Block

~UWC_TOKEN_START~1254808298013~UWC_TOKEN_END~ export GLOBUS_TCP_PORT_RANGE=40000,41000


e.g. You could set up:

Code Block

"/DC=org/DC=doegrids/OU=people/CN=Jason Cournoyer 939022" jcournoyer
"/DC=org/DC=doegrids/OU=People/CN=Sridhar Gullapalli 94604" sridhar
"/DC=org/DC=doegrids/OU=People/CN=Tiberiu Stef-Praun 764752" ww-user
"/DC=org/DC=doegrids/OU=people/CN=Stephen Norris Hookway 665012" ww-user


If you have a firewall then you will have to open access to the following ports from external IP hosts listed on the wiki page above.

Code Block

Table: Unix Port Numbers used by Grid software

Port 	service
22 	SSH and GSISSH over tcp
80 	HTTP server

443/8443	HTTPS server
636 	LDAP over SSL
1024 	GridFTP data return
2119 	Globus GRAM resource manager (Gatekeeper, tcp)
2811 	GridFTP contact (tcp)
7512 	MyProxy server
8080 	Web cache server