Running Jobs on the AHPCC SP/2

Table of Contents

• Running Interactively
• Running Batch Jobs
  • LoadLeveler/Maui Scheduler:
  • Job Command Files
  • Commands
  • Tips on Using LoadLeveler
• HPCERC SP2 Features

Running Interactively

Interactive jobs on the AHPCC SP2 can only run on the interactive SP2 nodes (fr1n05-08).

1. Login to an interactive SP2 node using 'ssh'
2. 
       ssh fr1n05
    
3. Compile code if necessary:
4. 
       mpcc filename.c or mpxlf filename.f -qnolm
            (Compiles MPI programs for C or Fortran respectively)
    
5. Set necessary environment variables:
• setenv MP_PROCS n  (when using n processors. Max - 4.)
• setenv MP_EUIDEVICE en0  (to use ethernet for communication)
• setenv MP_EUILIB ip  (to use IP protocol for communication)
• setenv MP_RMPOOL 0 (to specify Resource Manager pool)
6. Run program

Tips:

1. Make sure the interactive node names are in your .rhosts file
2. 
   fr1n05
   fr1n06
   fr1n07
   fr1n08

[Back to TOP]

Running Batch Jobs

Batch jobs on the AHPCC SP2 must be submitted from one of the interactive SP2 nodes (fr1n05-08) using LoadLeveler/Maui Scheduler.

1. Login to the interactive SP2 nodes using 'ssh'
2. 
       ssh fr1n05
3. Compile code if necessary:
4. 
       mpcc filename.c or mpxlf filename.f -qnolm
            (Compiles MPI programs for C or Fortran respectively)
5. Create Job Command file (e.g., test_run.cmd)
6. Submit job using "llsubmit" command
7. 
       llsubmit test_run.cmd
8. View job status in queue using "llq" and wait for completion
9. 
       llq

[Back to TOP]

AHPCC SP/2 Node Policy

* Limit applies to jobs running under Loadleveler only.
Does not apply to idle jobs waiting in the queue.

** Job length can be extended for special requests

Using LoadLeveler/Maui Scheduler

LoadLeveler is a batch job scheduling application and program product of IBM.  Maui Scheduler uses LoadLeveler but allocates jobs more fairly and efficiently. It also includes additional commands which give the user more information on jobs in the queue.

Job Command Files:
Command files describe the requirements and parameters for the job to be executed such as; number of processors, input/output files needed, memory/hardware requirements, etc.)  Comment lines begin with '#'. Valid LoadLeveler keywords begin with '#@'. Any line after the '#@ queue' will be executed at run-time.

Below is an sample command file. The file executes a program named "test_run" which resides in /home/joebloe/sp2example. Email will notify the user when the job is complete. Standard error and standard output are captured in  files "test.err" and "test.out" respectively as described by the "error" and "output" keywords. The "class" line identifies the processor-pool name that the program will run in. Maui Scheduler only has one pool name - "batch".

The program has the following requirement: Adapter=="hps_user". This means the program must use the high-performance communication switch. View the other possible SP2 features that are available (RAM, disk space, etc.)

The environment variable MP_EUILIB supporting this requirement is set by "environment". Finally, the number of processors is set by min_processors and max_processors (the maximum number is currently 16.)

Sample job command file at AHPCC

# Example MPI / MPL job command file
#
#@ initialdir      = /home/joebloe/sp2example
#@ notification    = complete
#@ notify_user     = joebloe@arc.unm.edu
#@ error           = test.err
#@ output          = test.out
#@ class           = batch
#@ job_type        = parallel
#@ requirements    = (Adapter == "hps_user")
#@ min_processors  = 4
#@ max_processors  = 4
#@ environment     = MP_EUILIB=us
#@ queue
test_run

[Back to TOP]

Commands:

llsubmit  job_command_file

Submits a job into the job queue.

---

llq

Obtains information in the job queue

---

llcancel  job_id

Removes the job from the queue. The job is identified by its job_id which can be obtained from the llq command.

---

showq

Displays how many nodes are available for immediate use on the system.

---

showbf

Displays the actual job ordering under the new scheduler

AHPCC SP/2 Features

Adapter=="hps_user"

Memory==64

Memory==128

Memory==256

Feature=="1G_SCRATCH"

[Back to TOP]