Running Jobs on Roadrunner

1. Overview
2. Interactive jobs
1. Overview
2. Requesting nodes for interactive jobs
3. Running Myrinet interactive jobs
4. Running Ethernet interactive jobs
3. Batch jobs
1. Using PBS scripts to submit your job
2. Monitoring your job
3. Monitoring the queue
4. Removing your job from the queue

The Roadrunner Linux Supercluster runs the Portable Batch System (PBS) as a resource manager with the Maui Scheduler.

2.0 Interactive jobs

2.1 Overview

The interactive or login nodes are not actually part of the compute cluster. Please do not try to run your executable on the login nodes.

You run your executable interactively by first requesting interactive access to a compute cluster node. Your request for interactive access to a node or nodes will be scheduled and you will be automatically connected to one of the nodes when your request is processed. Once you are connected to one of the compute nodes you may run your job as detailed below.

2.2 Requesting nodes for interactive jobs

To request n nodes for your interactive job enter:

$ qsub –I –l nodes=n <pbs_script>

The default time limit for interactive jobs is 15 minutes. To increase the time limit for your interactive job use the walltime argument:

$ qsub –I –l nodes=n,walltime=hh:mm:ss <pbs_script>
 

where hh is hours, mm is minutes, and ss is seconds. For example

$ qsub -I nodes=2,walltime=01:00:00 <pbs_script>
 

will request two nodes for one hour of interactive use for <pbs_script>.

You will be notified that your interactive job request has been submitted and you will be asked to wait until the requested nodes have been allocated:

$ qsub -I -l nodes=1,walltime=10:10:10 <pbs_script>
qsub: waiting for job 251.ll03.alliance.unm.edu to start
qsub: job 251.ll03.alliance.unm.edu ready

Roadrunner Portable Batch System
Job Id: 251.ll03.alliance.unm.edu
Username: janeuser

[janeuser@l001 ~]$

After a short time (depending on the number of nodes and the time requested) you will be connected or logged into one of the allocated nodes in the compute cluster.

2.3 Myrinet interactive jobs

To run your executable using the Myrinet communication protocol follow these steps:

1. To run your executable using one processor per node enter:
   
   $ gmpiconf $PBS_NODEFILE
   $ mpirun <executable>
    
2. To run your executable using two processors per node enter
   
   $ gmpiconf2 $PBS_NODEFILE
   $ mpirun -np 2*n <executable>
   
   where n is the number of nodes you requested for your interactive job.
3. When you are finished running your job interactively release the nodes:
   
   $ exit
   
   You will be disconnected from the allocated compute node and connected again to the login node.

2.4 Ethernet interactive jobs

To run your executable using the Ethernet communication protocol follow these steps:

1. To run your executable using one processor per node enter:
   
   $ mpirun <executable>
    
2. To run your executable using two processors per node enter:
   
   $ mpirun -np 2*n <executable>
   
   where n is the number of nodes you requested for your interactive job.
3. When you are finished running your job interactively release the nodes:
   
   $ exit
    
4. You will be disconnected from the allocated compute node and connected again to the login node.

back to Top

3.0 Batch jobs

3.1 Using PBS scripts to submit your job

Portable Batch System (PBS).
 The options supported are:

• -l resource name=resource[,...]
• -o output filename
• -e error filename
• -v envname=value[,...]
• -S shell command

The resource names supported using the '-l' option are

• nodes=node number
• walltime=time in format days:hours:minutes:seconds
• ppn=processors per node (2 or 1)

Your PBS script should include a line beginning with "#PBS", after which you should include any of the supported options as listed above.

Important: If you are using the Myrinet protocol for message passing, your PBS script must execute the gmpiconf (for one processor per node) command or the gmpiconf2 (for two processors per node) command before running your job using mpirun.

Below is an example PBS script for submitting jobs on Roadrunner using the qsub command:

# sample PBS script for use with 'qsub -P' on Roadrunner
#
# supported options to PBS command 'qsub' include
# -l resourcename=resource[,...]
# -o output filename
# -e error filename
# -v envname=vale[,...]
# -S shell command
#
# supported resourcenames include
# nodes=number of nodes
# walltime=time in days:hours:minutes:seconds
# ppn=processors per node (2 or 1)
#
#PBS -l nodes=4,walltime=00:00:10:00,ppn=2 -e myerror -o myout
gmpiconf2 $PBS_NODEFILE # use for Myrinet and 2 ppn
#gmpiconf $PBS_NODEFILE # use for Myrinet and 1 ppn
mpirun -np 8 -machinefile $PBS_NODEFILE a.out

3.2 Monitoring your job

To monitor your job use the qstat command:

$ qstat <jobID>
 

where <jobID> is the job ID for your job returned when you submitted the job. For example:

For more detailed information about your job use the '-rn' option to the qstat command.

To list all jobs currently in the queue use the qstat command with no job ID:

$ qstat

3.3 Monitoring the queue

To see all currently active and idle jobs, along with the number of active nodes and processors use the qstat command:

$ qstat

3.4 Removing your job from the queue

To remove your job from the queue or cancel a running job use the qdel command:

$ qdel 252.rr03

To verify that your job was removed from the queue, use the qstat command:

$ qstat

back to Top