Condor is a mechanism that supports High Throughput Computing (HTC). High Throughput Computing supports a large amount of processing capacity sustained over a very long period of time. Condor, in focusing HTC, allows users to submit jobs which are then executed using idle cycles on machines otherwise not being used. Whenever the load on the machine exceeds a certain pre-defined level or the mouse or keyboard is activated, Condor will remove any processes on that machine and not schedule it further. Condor programs are linked with checkpoint code that allows them to be unloaded from one machine and restarted, from where they left off, on another. Probably the best way to describe how Condor implements HTC is to use the terms "bottom feeder" or "cycle stealing" system.
Condor supports both parallel as well as serial jobs, but currently the only parallel paradigm available is PVM. At this time, Condor is on the Linux cluster machines and PVM is not supported on these machines, because these machines are not dedicated to computation but are used for workshops and HPC@UNM staff computers during normal working hours.
Currently, Condor is running on the HPC@UNM's Linux platforms including the Trucha Cluster. Condor can be implemented on other platforms and operating systems (AIX, Sun, HP, SGI), including NT. At this time, we are limiting our Condor pool to the Linux machines, but may in the future expand it to other machines.
The Condor version currently (as of 6/16/03) installed at HPC@UNM is the production version 6.4.7. This version will be updated when appropriate.
Currently, the HPC@UNM policy is to leave Condor running all the time. As Condor introduces very little overhead in a machine it has a minimal impact on machines being used for other purposes. When any Linux Cluster machine is in dedicated use mode, Condor may be shut down on that machines (for example, many of these machines are used for presenting workshops and are placed in dedicated mode for such purposes). For Condor job submission and job preparation, the Linux control workstation lcws.hpc.unm.edu will be available at all times (barring, of course, the need to bring it down for maintenance of the Linux cluster).
Condor jobs can be submitted in two ways: Standard or Vanilla; that is, the HPC@UNM Condor flock supports these two universes. Specifying these universes is shown below.
Standard Condor jobs must be linked using the condor_compile command. Recompilation is not necessary, only relinking. This installs the checkpointing code required for a standard job. The condor_compile command can be used on the final link step on any of the publicly available HPC@UNM Linux boxes. lcws.hpc.unm.edu has a full Condor install, and can be used to execute all Condor commands such as condor_compile and condor_submit when used in makefiles, scripts, etc. (see the man pages).
Vanilla Condor jobs do not use the Condor remote system or checkpoint services, but can be submitted and scheduled as a regular Condor job. Any Unix executable or script may be submitted to Condor as a vanilla job. They will not, however, perform remote system calls (not really a problem in our environment... see the man pages) or checkpoint. This means when a job is kicked off of a machine, it must be started over from scratch.
Condor batch jobs on the AHPCC Condor flock must be submitted from one of the Turing machines (workshop machines sucu as turing01, turing02, ..., turing12), other Linux machines such as rcde.., or the Linux control workstation lcws using the Condor submit command condor_submit (see the manual page for condor_submit).
Command files describe the requirements and parameters for the job to be executed, such as; input/output, error, and log files needed, memory/hardware requirements, etc. Comment lines begin with '#'. Valid Condor keywords are any initial word (no blanks) followed by an equal (=) sign.
Below is an sample Condor command file. The file executes a program named "test_run" which resides in the directory "/home/joebloe/condor_example". The name of the executable program is specified in line with keyword "Executable". Email will always notify the user about the status of the job. This is specified in the lines with the keyword "Notification" with the option "always" and with the keyword "Notify_user" with the option "joebloe@ahpcc.unm.edu". Standard error, standard output, and the job log file are captured in files "test.err", "test.out", and "test.log" respectively as described by the "Error", "Output", and "Log" keywords. The "Universe" line identifies the type of universe for the Condor job (either vanilla, standard, pvm, or scheduler). If the executable has command line arguments, then they are specified in the line with the keyword "Arguments". Finally, the directory in which the standard input, output, and log files are to be placed is specified in the line with keyword "InitialDir".
The program may have further requirements such as minimum RAM, minimum scratch disk storage, and so on. These and essentially any arbitrary requirements that are specified by the characteristics of the machines pool can be specified in a line with the keyword "Requirements". In addition, the user can rank preferences for machines so that machines with the highest rank will be checked first to see if they are available to run the job. For example, the machine with the largest available RAM exceeding some limit can be specified. Please refer to the tutorial on requirements and rank or Condor manual for more details on the topics of specifying requirements and rank.
A tutorial giving and explaining more sophisticated Condor command files can be found here: . The examples for this tutorial can be found in the directory: . The tutorial assumes you have copied this directory to your home directory and called it "examples".
A sample Condor job command file at HPC@UNM is:
# Example Condor job command file
# (Lines beginning with # are comments)
# NOTE: words on the left side of an equal sign are
# not case sensitive, but some items such as file names are.
Job_Type = "job"
Universe = standard
Executable = /home/joebloe/condor_example/test_run
InitialDir = /home/joebloe/condor_example
Notification = complete
Notify_user = joebloe@hpc.unm.edu
Error = test.err
Output = test.out
Log = test.log
requirements = (Arch == "INTEL")
queue
For additional keyword information,
check out Condor's manual at HPC@UNM (HPC@UNM's
reference copy) or for the latest Condor manual from Wisconsin (Wisconsin's
reference copy).
condor_submit job_command_file
Submits a job into the job queue.
condor_q
Obtains information about jobs in the job queue
condor_rm job_id
Removes the specified job from the queue. The job is identified by its job-id which is made up from its cluster number and task identifier which can be obtained from the condor_q command.
condor_status
Displays how many machines are available for immediate use on the system.
condor_history
Displays a history of your jobs, indicating whether they are still running, completed, or were cancelled.
The Wisconsin Condor manual at: http://www.cs.wisc.edu/condor/manual/v6.4/ is the ultimate source; make sure you are referencing the version of the manual for our implementation. At HPC@UNM, we are currently using Condor production version 6.4.7.
The man pages for the various commands
are available at: http://www.cs.wisc.edu/condor/manual/v6.4/9_Command_Reference.html