Running MPI jobs
MPI can only be run on the Black Bear compute nodes through PBS.
You can use the "mpirun" script to run an MPI job. The script has
the following syntax
mpirun -np n -machinefile $PBS_NODESFILE full_path/executable
-np n - Number of nodes to start
program on
full_path/executable - Executable
including full path name
Using the Batch Scheduler
(PBS)
Black Bear uses the Portable Batch System (PBS) to manage parallel
job scheduling. Users can run batch or interactive jobs on the Black
Bear Supercluster through PBS. Batch jobs are described in command
files which are submitted to PBS for execution.
PBS Queue Limits
|
| Maximum Job Length*: |
6 hours |
* Job length can be extended for special requests.
Please contact help@ahpcc.unm.edu
PBS Command Files
The PBS job command file defines the number of nodes you are requesting
and contains the shell commands you wish to be executed. One or
two processes per node can be started. Some samples are listed below
for a job to run the executable complete_path/test_prog
on N nodes
(for a user using a (t)csh style shell):
For Myrinet jobs (one process per node)
#PBS -l nodes=N
# source to get environment variables
# cshrc_bb - for (t)csh users
# bashrc_bb - for (ba)(k)sh users
# uncomment the proper line based on your shell
source $HOME/BLACKBEAR/cshrc_bb
#. $HOME/BLACKBEAR/bashrc_bb
gmpiconf $PBS_NODEFILE
mpirun -np N -machinefile $PBS_NODEFILE <full path to your executable>
For MPI jobs to run using Myrinet, the "gmpiconf" command must
be used to create a configuration file ($HOME/.gmpi/conf). To
run two processes per node, use the gmpiconf2 command and
double the number of processes (-np 2*n) for the mpirun
command as shown below:
For Myrinet jobs (two processes per node)
gmpiconf2 $PBS_NODEFILE
mpirun -np 2*N -machinefile $PBS_NODEFILE complete_path/test_prog
For Ethernet jobs (one process per node)
mpirun -np N -machinefile $PBS_NODEFILE complete_path/test_prog
For Ethernet jobs (two processes per node)
#PBS -l nodes=N
mpirun -np 2*N -machinefile $PBS_NODEFILE complete_path/test_prog
The "#PBS -l nodes=N" line defines how many nodes are being
requested. The PBS environment variables which are used in the script
above are defined below. These variables are set when PBS allocates
nodes for a user.
Example script (sample.cmd): 32-node Myrinet using two
processes per node:
#PBS -l nodes=16
# source to get environment variables
# cshrc_bb - for (t)csh users
# bashrc_bb - for (ba)(k)sh users
# uncomment the proper line based on your shell
source $HOME/BLACKBEAR/cshrc_bb
#. $HOME/BLACKBEAR/bashrc_bb
gmpiconf2 $PBS_NODEFILE
mpirun -np 32 /home/cashmont/test.exe
PBS Environment Variables
- PBS_O_WORKDIR - user's current directory when "qsub"
command was issued
- PBS_NODEFILE - file containing the list of nodes
the user was allocated
Submiting jobs to PBS
qsub [flags] (script_name)
Submit a batch job listed in scriptname.pbs file
qsub scriptname.pbs
Displaying the queue contents and status
qstat [flags]
i.e., qstat -s
(displays status and elapsed time of job)
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
2534.bb nestor12 rr STDIN 23483 16 -- -- 06:00 R 01:04
Job started on Fri Jul 09 at 15:53
2554.bb corona rr STDIN 19704 2 -- -- 06:00 R 00:44
Job started on Fri Jul 09 at 16:14
2557.bb corona rr STDIN 22012 2 -- -- 06:00 R 00:42
Job started on Fri Jul 09 at 16:15
2559.bb nestor12 rr alltoall.p -- 64 -- -- 06:00 Q --
Not Running: Not enough of the right type of nodes are available
2560.bb nestor12 rr alltoall12 -- 64 -- -- 06:00 Q --
Not Running: Not enough of the right type of nodes are available
- Job ID: ID for users' job in queue
- Username: job owner
- Jobname: name of script submitted to PBS or STDIN
if user is in an interactive session
- NDS: number of nodes allocated
- Time: maximum wall clock run time
- S (State): R - Running; Q - Waiting in queue; E -
Job is in error state
- Time (last column): job's elapsed time
Remove job from queue
qdel (job_id)
where job_id can be obtained from the queue listing using "qstat".
Output and Error files
Standard error and output are contained in two files with the names:
Jobname.eJob_ID
Jobname.oJob_ID
using the Jobname and Job_ID from your job in the queue.
Running Interactive Jobs
You can initiate an interactive session on Black Bear nodes through
PBS. This is the only case where a user has direct access to a compute
node.
Initiate an interactive session using n nodes
qsub -I -l nodes=n
Once an interactive session is granted, you are placed in your
home directory on one of the nodes you were allocated. Execute
the code as described in MPIRUN section.
|
