CARC web archive · Archived copy of hpc.unm.edu/computing/blackbear/ug/index.html as captured by the Internet Archive on 25 July 2003. Links to pages that were not saved here go to the Wayback Machine. Wayback copy · All pages in this snapshot · Web archive index

Black Bear Linux Supercluster

System Facts:

  • Nodes: 16
  • CPU: Dual Intel Pentium III 550 MHz
  • Cache: 512Kb ECC RAM
  • RAM: 512 MB SDRAM
  • Hard Disk: 9 GB SCSI II
  • Network (control): 10/100 BaseT
  • Network (data): Myrinet [GM v.1.1.3]
  • OS: VA Red Hat 6.0.4
  • Kernel: Linux 2.2.12smp

General User Information

1. Access
2. Your Account
3. Environment
4. Software:
     Compilers
     Programming Tools
     Programming for MPI
     Mathematic and Scientific Libraries
5. Running Jobs
     Selecting MPI Libraries
     Compiling
     Running MPI programs
     Using the Batch Scheduler (PBS)
     Submitting Jobs to PBS
     Running Interactive Jobs
6. Storage

Access

The Black Bear Supercluster is accessed through one machine. This front-end system, named blackbear.ahpcc.unm.edu, is intended for code development and submitting jobs to the Black Bear compute nodes. 

Logging in:
Use a secure login utility such as secure shell (ssh) to login to the Black Bear systems. 
ssh blackbear.ahpcc.unm.edu
ssh bb.ahpcc.unm.edu
ssh osonegro.ahpcc.unm.edu

   

Your Account

You can access Black Bear using your regular AHPCC account. If you don't have an AHPCC account, you can apply for one via: http://santafe.hpcerc.unm.edu/irs/
 
   

Environment

Many environment variables needed for compilers and libraries are defined in certain config files. Users can run the /nfs/linux/BLACKBEAR/scripts/setup_env.bb script to setup your environment to run on Black Bear.
  • setup_env.bb (script to setup your environment to run on Black Bear)
  • test.pbs (sample submit file) 
  • rhosts (contains the hostnames of the Black Bear nodes)
  • cshrc_bb (for csh and tcsh users)
  • bashrc_bb (for ksh and bash users)
   

Software
Black Bear has a number of compilers (C, C++, Fortran 77/90, and HPF) including the GNU, NAGf95 and Portland Group suite. The tables below list the specific compiler commands and links to documentation when available. 
Compilers
C C++ Fortran77 Fortran90/95 High. Perf. Fortran
(HPF)
gcc
pgcc		 g++
pgCC		 g77
pgf77		 f90
pgf90
f95		 pghpf
Included in the Portland Group suite are two useful programming tools. The source code debugger and performance profiler is documented in the on-line PGI manual. 
Programming Tools
Source Code Debugger Performance Profiler
pgdbg pgprof
Compile scripts are available for compiling MPI code. These scripts automatically link in all MPI libraries and include files. 
Compiling for MPI
C C++ Fortran77 Fortran90
mpicc mpiCC mpif77 mpif90
Mathematic and Scientific Libraries
Linear Algebra
  BLAS 
(Basic Linear Algebra Subroutines)		 /usr/lib/libblas.a
/usr/pgi/linux86/lib/libblas.a* 
* (for PGI compilers only)
  LAPACK /usr/lib/liblapack.a
/usr/pgi/linux86/lib/liblapack.a* 
* (for PGI compilers only)
FFT
  FFTW /usr/local/lib/libfftw_mpi.a
/usr/local/lib/librfftw_mpi.a
 
   
   

Running Jobs

Selecting MPI Libraries

There are four MPI libraries on the Black Bear Supercluster, two using Myrinet (for PGI and GNU compilers) and two using Ethernet (for PGI and GNU). Select a library by setting the MPIHOME environment variable. Refer to your .cshrc file for examples. 

Use the "which" command to verify your compile library. For example:

% source $HOME/BLACKBEAR/cshrc_bb

% which mpicc
/usr/parallel/mpich-eth.gnu/bin/mpicc
shows that you are using Ethernet with GNU compilers. And, 

% which mpicc
/usr/parallel/mpich-gm.pgi/bin/mpicc
shows that you are using Myrinet with PGI compilers. 

Compiling

Compile and link C code with MPI libraries 
% mpicc -o exec [options] filename(s).c

Compile and link C++ code with MPI libraries 
% mpiCC -o exec [options] filename(s).C

Compile and link Fortran77 code with MPI libraries 
% mpif77 -o exec [options] filename(s).f

Compile and link Fortran90 code with MPI libraries 
% mpif90 -o exec [options] filename(s).f90

Running MPI jobs

MPI can only be run on the Black Bear compute nodes through PBS. You can use the "mpirun" script to run an MPI job. The script has the following syntax 
mpirun -np n -machinefile $PBS_NODESFILE full_path/executable
-np n  - Number of nodes to start program on 
full_path/executable  - Executable including full path name

Using the Batch Scheduler (PBS)

Black Bear uses the Portable Batch System (PBS) to manage parallel job scheduling. Users can run batch or interactive jobs on the Black Bear Supercluster through PBS. Batch jobs are described in command files which are submitted to PBS for execution. 

PBS Queue Limits
Maximum Job Length*: 6 hours
* Job length can be extended for special requests.
Please contact help@hpcerc.unm.edu

PBS Command Files

The PBS job command file defines the number of nodes you are requesting and contains the shell commands you wish to be executed. One or two processes per node can be started. Some samples are listed below for a job to run the executable complete_path/test_prog on N nodes 
(for a user using a (t)csh style shell): 
For Myrinet jobs (one process per node) 
#PBS -l nodes=N
# source to get environment variables
# cshrc_bb - for (t)csh users
# bashrc_bb - for (ba)(k)sh users 
# uncomment the proper line based on your shell
source $HOME/BLACKBEAR/cshrc_bb
#. $HOME/BLACKBEAR/bashrc_bb
gmpiconf $PBS_NODEFILE
mpirun -np N -machinefile $PBS_NODEFILE <full path to your executable>
For MPI jobs to run using Myrinet, the "gmpiconf" command must be used to create a configuration file ($HOME/.gmpi/conf). To run two processes per node, use the gmpiconf2 command and double the number of processes (-np 2*n) for the mpirun command as shown below: 

For Myrinet jobs (two processes per node) 

gmpiconf2 $PBS_NODEFILE
mpirun -np 2*N -machinefile $PBS_NODEFILE complete_path/test_prog
For Ethernet jobs (one process per node) 
mpirun -np N -machinefile $PBS_NODEFILE complete_path/test_prog
For Ethernet jobs (two processes per node) 
#PBS -l nodes=N
mpirun -np 2*N -machinefile $PBS_NODEFILE complete_path/test_prog
The "#PBS -l nodes=N" line defines how many nodes are being requested. The PBS environment variables which are used in the script above are defined below. These variables are set when PBS allocates nodes for a user. 
Example script (sample.cmd): 32-node Myrinet using two processes per node: 
#PBS -l nodes=16
# source to get environment variables
# cshrc_bb - for (t)csh users
# bashrc_bb - for (ba)(k)sh users 
# uncomment the proper line based on your shell
source $HOME/BLACKBEAR/cshrc_bb
#. $HOME/BLACKBEAR/bashrc_bb
gmpiconf2 $PBS_NODEFILE
mpirun -np 32 /home/cashmont/test.exe

PBS Environment Variables

  • PBS_O_WORKDIR - user's current directory when "qsub" command was issued
  • PBS_NODEFILE - file containing the list of nodes the user was allocated
Submiting jobs to PBS
qsub [flags] (script_name) 
Submit a batch job listed in scriptname.pbs file 
qsub scriptname.pbs 
Displaying the queue contents and status 
qstat [flags]
i.e., qstat -s (displays status and elapsed time of job)  
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
2534.bb         nestor12 rr       STDIN       23483  16  --    --  06:00 R 01:04
   Job started on Fri Jul 09 at 15:53
2554.bb         corona   rr       STDIN       19704   2  --    --  06:00 R 00:44
   Job started on Fri Jul 09 at 16:14
2557.bb         corona   rr       STDIN       22012   2  --    --  06:00 R 00:42
   Job started on Fri Jul 09 at 16:15
2559.bb         nestor12 rr       alltoall.p    --   64  --    --  06:00 Q   --
   Not Running: Not enough of the right type of nodes are available
2560.bb         nestor12 rr       alltoall12    --   64  --    --  06:00 Q   --
   Not Running: Not enough of the right type of nodes are available
  • Job ID: ID for users' job in queue
  • Username: job owner
  • Jobname: name of script submitted to PBS or STDIN if user is in an interactive session
  • NDS: number of nodes allocated
  • Time: maximum wall clock run time 
  • S (State): R - Running; Q - Waiting in queue; E - Job is in error state
  • Time (last column): job's elapsed time
Remove job from queue
qdel (job_id)

where job_id can be obtained from the queue listing using "qstat". 

Output and Error files

Standard error and output are contained in two files with the names:  
      Jobname.eJob_ID
      Jobname.oJob_ID
using the Jobname and Job_ID from your job in the queue. 

Running Interactive Jobs

You can initiate an interactive session on Black Bear nodes through PBS. This is the only case where a user has direct access to a compute node. 

Initiate an interactive session using n nodes 
qsub -I -l nodes=n

Once an interactive session is granted, you are placed in your home directory on one of the nodes you were allocated. Execute the code as described in MPIRUN section.
   

Storage

Approximately 3 GB of scratch space is available on each Black Bear compute node in /scratch. 
/scratch is for TEMPORARY storage only - All files in /scratch will be deleted when the job has finished
   

bsp;  19704   2  --    --  06:00 R 00:44    Job started on Fri Jul 09 at 16:14 2557.bb         corona   rr       STDIN       22012   2  --    --  06:00 R 00:42    Job started on Fri Jul 09 at 16:15 2559.bb         nestor12 rr       alltoall.p    --   64  --    --  06:00 Q   --    Not Running: Not enough of the right type of nodes are available 2560.bb         nestor12 rr       alltoall12    --   64  --    --  06:00 Q   --    Not Running: Not enough of the right type of nodes are available Remove job from queue
qdel (job_id)

where job_id can be obtained from the queue listing using "qstat". 

Output and Error files

Standard error and output are contained in two files with the names:  
      Jobname.eJob_ID
      Jobname.oJob_ID
using the Jobname and Job_ID from your job in the queue. 

Running Interactive Jobs

You can initiate an interactive session on Black Bear nodes through PBS. This is the only case where a user has direct access to a compute node. 

Initiate an interactive session using n nodes 
qsub -I -l nodes=n

Once an interactive session is granted, you are placed in your home directory on one of the nodes you were allocated. Execute the code as described in MPIRUN section.    

Storage

Approximately 3 GB of scratch space is available on each Black Bear compute node in /scratch.