Access

All systems are accessed through their own machine. This front-end system, named roadrunner.alliance.unm.edu, is intended for code development and submitting jobs to the Roadrunner compute nodes.

Logging in:
Use a secure login utility such as secure shell (ssh) to login to the system.
for example: ssh roadrunner.alliance.unm.edu

FTP:
Use a secure file transfer utility such as secure copy (scp) to transfer files to the Roadrunner systems.
scp file rr_user@roadrunner.alliance.unm.edu:dest_file        -> uploads a local file to Roadrunner

Users can FTP their files from Roadrunner.
 

Your Account

You can change your password using the "yppasswd" command. You will be prompted for your current password in order to change it.
yppasswd

Changing your default shell is similar to changing your password. Use the "ypchsh" command and you will be prompted for your password and then the new shell you wish to use (including full path - e.g., /bin/tcsh)
ypchsh
 

Environment

Many environment variables needed for compilers and libraries are defined in your .cshrc. Users can copy the current default files from /etc/skel.

• .cshrc (for Csh and Tcsh users)

Software

Included in the Portland Group suite are two useful programming tools. The source code debugger and performance profiler is documented in the on-line PGI manual.

Compile scripts are available for compiling MPI code. These scripts automatically link in all MPI libraries and include files.

Running Jobs

Selecting MPI Libraries

There are four MPI libraries on the Roadrunner Supercluster, two using Myrinet (for PGI and GNU compilers) and two using Ethernet (for PGI and GNU). Select a library by setting the MPIHOME environment variable. Refer to your .cshrc file for examples.

Use the "which" command to verify your compile library. For example,

% which mpicc
/usr/parallel/mpich-eth.gnu/bin/mpicc
shows that you are using Ethernet with GNU compilers. And,

% which mpicc
/usr/parallel/mpich-gm.pgi/bin/mpicc
shows that you are using Myrinet with PGI compilers.

Compiling

Compile and link C code with MPI libraries
% mpicc -o exec [options] filename(s).c

Compile and link C++ code with MPI libraries
% mpiCC -o exec [options] filename(s).C

Compile and link Fortran77 code with MPI libraries
% mpif77 -o exec [options] filename(s).f

Compile and link Fortran90 code with MPI libraries
% mpif90 -o exec [options] filename(s).f90

Running MPI jobs

MPI can only be run on the Roadrunner compute nodes through PBS. You can use the "mpirun" script to run an MPI job. The script has the following syntax

mpirun -np n full_path/executable

-np n  - Number of nodes to start program on
full_path/executable  - Executable including full path name

Using the Batch Scheduler (PBS)

Roadrunner uses the Portable Batch System (PBS) to manage parallel job scheduling. Users can run batch or interactive jobs on the Roadrunner Supercluster through PBS. Batch jobs are described in command files which are submitted to PBS for execution.

* Job length can be extended for special requests.
Please contact help@alliance.unm.edu

PBS Command Files

The PBS job command file defines the number of nodes you are requesting and contains the shell commands you wish to be executed. One or two processes per node can be started. Some samples are listed below for a job to run the executable complete_path/test_prog on N nodes: 

For Myrinet jobs (one process per node)

#PBS -l nodes=N
gmpiconf $PBS_NODEFILE
mpirun complete_path/test_prog

For MPI jobs to run using Myrinet, the "gmpiconf" command must be used to create a configuration file ($HOME/.gmpi/conf). To run two processes per node, use the gmpiconf2 command and double the number of processes (-np 2*n) for the mpirun command as shown below:

For Myrinet jobs (two processes per node)

#PBS -l nodes=N
gmpiconf2 $PBS_NODEFILE
mpirun -np 2*N complete_path/test_prog

For Ethernet jobs (one process per node)

#PBS -l nodes=N
mpirun complete_path/test_prog

For Ethernet jobs (two processes per node)

#PBS -l nodes=N
mpirun -np 2*N complete_path/test_prog

The "#PBS -l nodes=N" line defines how many nodes are being requested. The PBS environment variables which are used in the script above are defined below. These variables are set when PBS allocates nodes for a user.

Example script (sample.cmd): 32-node Myrinet using two processes per node:

#PBS -l nodes=32
gmpiconf2 $PBS_NODEFILE
mpirun -np 64 /users/wilson/solve_run.exe

PBS Environment Variables

• PBS_O_WORKDIR - user's current directory when "qsub" command was issued
• PBS_NODEFILE - file containing the list of nodes the user was allocated

Submiting jobs to PBS
qsub [flags] (script_name)

Submit a batch job listed in scriptname.pbs file
qsub scriptname.pbs

Displaying the queue contents and status
qstat [flags]

i.e., qstat -s (displays status and elapsed time of job)