Child pages
  • Grid Setup

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

http://osg.ivdgl.org/twiki/bin/view/ReleaseDocumentation/CEFirewalls provides a very nice description of the firewalls and ports that need to be opened. It lists incoming and outgoing connection details. Note: You can specify PORT ranges for incoming connections (at least a hundered) plus the list of standard ports required for grid services. I would say to start we are setting up a small grid.

Once we have Globus running and tested, we should have a list of hosts on the TANGRAM Grid, with IP address and DNS, to allow Firewalls to be used effectively. We plan to run some monitoring software providing live status of the TANGRAM Grid.

As a group we could agree on a TCP PORTRANGE for Globus Connections.

Network Firewall Ports

These ports are to be opened bi-directional

  1. GRIDFTP 2811
  2. GATEKEEPER 2119
  3. GSISSH 40022
  4. GLOBUS TCP PORT RANGE = 40000,41000 or more if possible.
  5. Allegro Graph Server port = 4567
  6. MYSQL Server port = 3306
  7. HTTP ports (outgoing to connect to various services) 80, 8080, 8443, 443 etc
  8. GANGLIA ports 8655, 8649
  9. DC port ?
  10. PC port ?

...

  • Modify the gmond.conf file for your setup.
    • Edit the following section and put values for your cluster.
    • 2.x versions the following entries need to be changed name, owner, url, mcast, setuid.
      Panel

      name "Viz"
      owner "ISI / CGT"
      url "http://wind.isi.edu/ganglia/"
      mcast_if eth0
      setuid ganglia

      • 3.x versions do following
        Panel

        cluster {
        name = "Windward"
        owner = "ISI / CGT"
        latlong = "N30.0 W122.23"
        url = "http://wind.isi.edu/ganglia"
        }

  • edit the udp_send_channel section and change the mcast_join hostname/ip to be the hostname where your gmetad daemon is running.
    Panel

    udp_send_channel {
    mcast_join = wind.isi.edu
    port = 8649
    ttl = 1
    }

...

  • Edit the /etc/gmetad.conf file
    • 2.x the following entries need to be changed
      Panel

      data_source "<Clustername>" localhost
      gridname "<your organization>"
      authority "http://wind.isi.edu/ganglia/"
      trusted_hosts 127.0.0.1 wind.isi.edu <yourhostname or ip>
      setuid_username "ganglia"

...

Here is an example of what is displayed

Code Block

root@ttwo:/local/software/grid/condor/src/condor-7.1.3# condor_status

Name               OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime

slot1@ttwo.stdc.co LINUX      INTEL  Unclaimed Idle     0.000  1013  0+00:05:04
slot2@ttwo.stdc.co LINUX      INTEL  Unclaimed Idle     0.000  1013  0+00:05:05

{panel}
                     Total Owner Claimed Unclaimed Matched Preempting Backfill
{panel}

         INTEL/LINUX     2     0       0         2       0          0        0

{panel}
               Total     2     0       0         2       0          0        0
{panel}
root@ttwo:/local/software/grid/condor/src/condor-7.1.3#

root@ttwo:/local/software/grid/condor/src/condor-7.1.3# condor_q


-- Submitter: ttwo.stdc.com : <98.173.240.82:59254> : ttwo.stdc.com
{panel}
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
{panel}


...

  • To start with
    • Write a file called globus-gatekeeper in /etc/xinetd.d directory
    • Replace <$TANGRAM_ROOT_DIR> with the value of the environment variable that you set
Code Block

service globus-gatekeeper
{panel}
   {
      socket_type  = stream
      protocol     = tcp
      wait         = no
      user         = root
      server       = <$TANGRAM_ROOT_DIR>/globus/default/sbin/globus-gatekeeper
      server_args  = -conf <$TANGRAM_ROOT_DIR>/globus/default/etc/globus-gatekeeper.conf
      disable      = no
      env          = LD_LIBRARY_PATH=<$TANGRAM_ROOT_DIR>/globus/default/lib
      env         += GLOBUS_LOCATION=<$TANGRAM_ROOT_DIR>/globus/default
     env         += GLOBUS_TCP_PORT_RANGE=40000,41000
   }
{panel}
    • Write a file called gridftp in the same directory
    • Replace <$TANGRAM_ROOT_DIR> with the value of the environment variable that you set
Code Block

service gridftp
{panel}
    {
            instances               = 100
            socket_type             = stream
            wait                    = no
            user                    = root
            server                  = <$TANGRAM_ROOT_DIR>/globus/default/sbin/globus-gridftp-server
            server_args             = -i -d info -l <~UWC_TOKEN_START~1254808297989~UWC_TOKEN_END~TANGRAM_ROOT_DIR>/globus/default/var/gridftp.log
            log_on_success         += DURATION USERID
            log_on_failure         += USERID
            nice                    = 10
            disable                 = no
            env                    += GLOBUS_LOCATION=<$TANGRAM_ROOT_DIR>/globus/default
            env                    += PATH=<~UWC_TOKEN_START~1254808297991~UWC_TOKEN_END~TANGRAM_ROOT_DIR>/globus/default/bin:<~UWC_TOKEN_START~1254808297992~UWC_TOKEN_END~TANGRAM_ROOT_DIR>/globus/default/sbin
            env                    += LD_LIBRARY_PATH=<~UWC_TOKEN_START~1254808297993~UWC_TOKEN_END~TANGRAM_ROOT_DIR>/globus/default/lib
           env         += GLOBUS_TCP_PORT_RANGE=40000,41000
    }
{panel}

...

  • source $GLOBUS_LOCATION/etc/globus-user-env.sh
  • Run the command grid-cert-request -ca 0683a0c5 -host "fullhostname-including-domain". e.g.
    Panel

    grid-cert-request -ca 0683a0c5 -host "mymachine.mysite.com"

...

  • source $GLOBUS_LOCATION/etc/globus-user-env.sh
  • Run the command grid-cert-request -ca 0683a0c5 -cn "Your Full Name + 6 digitpin" . It will prompt you to specify a password. Put your password... e.g.
    Panel

    grid-cert-request -ca 0683a0c5 -cn "Gaurang Mehta 123456"

...

GLOBUS_TCP_PORT_RANGE 40000,41000

Code Block

sukhna 32% grid-proxy-init
Your identity: /DC=org/DC=doegrids/OU=People/CN=Karan Vahi 476301
Enter GRID pass phrase for this identity:
Creating proxy ................................. Done
Your proxy is valid until: Sat Oct 25 03:03:12 2008


Testing the grid ftp server

sukhna 33% telnet ttwo.stdc.com 2811
Trying 98.173.240.82...
Connected to ttwo.stdc.com.
Escape character is '^]'.
220 ttwo.stdc.com GridFTP Server 2.8 (gcc32, 1217607445-63) [Globus Toolkit 4.0.8] ready.



Testing the jobmanager

sukhna 34%
sukhna 34% globusrun -a -r ttwo.stdc.com

GRAM Authentication test successful
sukhna 35%

sukhna 41% setenv GLOBUS_TCP_PORT_RANGE 40000,41000
sukhna 42% globus-job-run ttwo.stdc.com /bin/date
Fri Oct 24 18:09:43 EDT 2008


sukhna 43% globus-job-run ttwo.stdc.com/jobmanager-condor /bin/date
Fri Oct 24 18:11:05 EDT 2008
sukhn


{panel}
 gsissh -p 40022 ttwo.stdc.com
{panel}
Warning: No xauth data; using fake authentication data for X11 forwarding.
Linux ttwo 2.6.24-etchnhalf.1-686 #1 SMP Mon Jul 21 11:17:43 UTC 2008 i686

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
/usr/bin/X11/xauth:  creating new authority file /home/sr/.Xauthority
sr@ttwo:~~UWC_TOKEN_START~1254808298002~UWC_TOKEN_END~ exit

...

NOTE: you will issue the commands from your machine containing the VDT installation.
(the following is for BASH shells, others CSH/TCSH should change these accordingly):

Code Block

~UWC_TOKEN_START~1254808298003~UWC_TOKEN_END~ export GLOBUS_LOCATION=<path_to_globus>
~UWC_TOKEN_START~1254808298004~UWC_TOKEN_END~ source ~UWC_TOKEN_START~1254808298005~UWC_TOKEN_END~GLOBUS_LOCATION/etc/globus-user-env.sh

You need to have the credentials in the right place, and initialized

Code Block

$ mkdir ~/.globus

$ cp usercert.pem ~/.globus
$ cp userkey.pem ~/.globus
$ grid-proxy-init

You are ready to issued grid commands

Test the ISI Grid

VIZ

  • Check if you can authenticate
    Code Block
    
    ~UWC_TOKEN_START~1254808298010~UWC_TOKEN_END~ globusrun -a -r viz-login.isi.edu
    
  • Check if you can run a job
    Code Block
    
    $ globus-job-run viz-login.isi.edu /bin/date
    
  • Check if you can transfer a file
    Code Block
    
    ~UWC_TOKEN_START~1254808298012~UWC_TOKEN_END~ globus-url-copy -dbg -vb file:///tmp/sometemp file gsiftp://viz-login.isi.edu/tmp/mynewtestfile
    

...

Most of the time you will get the ill-famed

Code Block

GRAM Job submission failed because the job manager failed to open stderr (error code 74)

This is caused by the fact that you are firewalled, and you can address these in the following way: one trick you can use is to specify that your machine expects replies
on a specific range of ports:

Code Block

~UWC_TOKEN_START~1254808298013~UWC_TOKEN_END~ export GLOBUS_TCP_PORT_RANGE=40000,41000

...

e.g. You could set up:

Code Block

"/DC=org/DC=doegrids/OU=people/CN=Jason Cournoyer 939022" jcournoyer
"/DC=org/DC=doegrids/OU=People/CN=Sridhar Gullapalli 94604" sridhar
"/DC=org/DC=doegrids/OU=People/CN=Tiberiu Stef-Praun 764752" ww-user
"/DC=org/DC=doegrids/OU=people/CN=Stephen Norris Hookway 665012" ww-user
.
.
.

...

If you have a firewall then you will have to open access to the following ports from external IP hosts listed on the wiki page above.

Code Block

Table: Unix Port Numbers used by Grid software

Port 	service
22 	SSH and GSISSH over tcp
80 	HTTP server

443/8443	HTTPS server
636 	LDAP over SSL
1024 	GridFTP data return
2119 	Globus GRAM resource manager (Gatekeeper, tcp)
2811 	GridFTP contact (tcp)
7512 	MyProxy server
8080 	Web cache server

...