Submit Jobs on Pythia

Pythia employs the SLURM workload manager to prioritize jobs and allocated resources. At a minimum, SLURM must be told which account a user belongs to (--account) and what hardware is being requested (--partition).

Scheduler Configuration

Slurm Accounts

All users must belong to an account before being able to access the compute nodes. View your available accounts by copy-pasting the literal command below:

sacctmgr show association where user=${USER} format=user,account

Note

If the above command does not return any accounts, please contact research.support@chicagobooth.edu to request access to Pythia

Slurm Partitions

To see a list of available partitions, use the sinfo command:

$ sinfo

Currently, Pythia is configured with the following partitions:

Partition	Nodes	Cores	Memory	GPU	Wall clock	GPU Type
interactive_l40s	Def: 1 Max: 1	Def: 1 Max: 64	Def: 16GB Max:750GB	Min:1 Max:8	Def: 2h Max: 2h	L40S
standard_l40s	Def: 1 Max: 1	Def: 1 Max: 64	Def: 16GB Max:750GB	Min:1 Max:8	Def: 4h Max: 3d	L40S
interactive_hopper	Def: 1 Max: 1	Def: 1 Max: 64	Def: 16GB Max: 2 TB	Min:1 Max:8	Def: 2h Max: 2h	H100 and H200
standard_hopper	Def: 1 Max: 1	Def: 1 Max: 64	Def: 16GB Max: 2 TB	Min:1 Max:8	Def: 4h Max: 3d	H100 and H200
long_hopper	Def: 1 Max: 1	Def: 1 Max: 64	Def: 16GB Max: 2 TB	Min:1 Max:8	Def: 3d Max: 14d	H100 and H200

The standard partitions are suitable for most batch jobs.

The interactive partitions are meant for light workloads (e.g. debugging, testing, short exploratory runs, etc.) that require user input.

The long_hopper partition is available for batch jobs that require more than 3 days to run.

Environment and Modules

Booth IT maintains a curated set of software programs commonly in use by Booth researchers. To facilitate setting up the computing environments, we use software modules. Below are some examples of software modules that are currently available on Pythia:

Python
Conda
Cuda
R
Matlab

Additionally, we provide cloud provider command line interface (CLI) to reach out to your cloud account from Pythia:

aws-cli
gcloudcli

Once logged in to Pythia, modules can be used in the following manner.

View the list of currently loaded modules: module list
View a list of the available software modules: module avail
Load a specific module: module load <module_name>
Unload a specific module: module unload <module_name>
Unload all loaded modules: module purge

Note

To access modules commands, you must be logged in to a compute node

Loading and unloading modules is typically done for every session. In addition, you may load modules inside a submission script. More detailed information for the module command can be found by typing man module.

# make sure you are on a compute node
$ srun --partition=interactive_l40s --account=<accountname> --gres=gpu:l40s:1 --pty bash --login

# list currently loaded modules
$ module list
No Modulefiles Currently Loaded.

# check which version of R would be used
$ which Python3
/usr/bin/python3

# find out what modules are available
$ module avail

------------------------- /apps/modulefiles/pythia ------------------------
conda/23.10
python/booth/3.12
R/4.3/4.3.2

# now, load a specific version of Python (3.12)
$ module load  python/booth/3.12

# verify that the module has been loaded
$ module list
Currently Loaded Modulefiles:
      1) python/booth/3.12

# check the version of Python3
$ which python3
/apps/python/RHEL8/3.12/3.12.9/bin/python3

# unload all of the currently loaded modules
$ module purge

# check that the version of R has reverted
$ which python3
/usr/bin/python3

Running Programs on Pythia

When connecting to Pythia, a user is directed to a login node (pytfe02 or pytfe03) The login node is useful for viewing your home directory or for submitting computational tasks on Pythia’s compute nodes via the sbatch command. The login node should never be used to directly run any computational tasks or to large datasets.

Submitting Batch Jobs

The sbatch command is the command most commonly used to request computing resources on Pythia.

Rather than specify all the options in the command line, users typically write a submission script that contains all the commands and parameters necessary to run the program on the cluster.

In a submission script, all Slurm parameters are declared with #SBATCH, followed by additional definitions.

Here is an example of a submission script:

submit.sh

#!/bin/bash

#---------------------------------------------------------------------------------
# Account information

#SBATCH --account=faculty-<PI BoothID>

#---------------------------------------------------------------------------------
# Resources requested

#SBATCH --partition=standard_l40s    #  standard_l40s, standard_hopper
#SBATCH --cpus-per-task=1            # number of CPUs requested (for parallel tasks)
#SBATCH --mem=2G                     # requested memory
#SBATCH --time=0-04:00:00            # wall clock limit (d-hh:mm:ss)
#SBATCH --gres=gpu:2
#---------------------------------------------------------------------------------
# Job specific name (helps organize and track progress of jobs)

#SBATCH --job-name=my_batch_job    # user-defined job name

#---------------------------------------------------------------------------------
# Print some useful variables

echo "Job ID: $SLURM_JOB_ID"
echo "Job User: $SLURM_JOB_USER"
echo "Num Cores: $SLURM_JOB_CPUS_PER_NODE"

#---------------------------------------------------------------------------------
# Load necessary modules for the job

module load <modulename>

#---------------------------------------------------------------------------------
# Commands to execute below...

<commands>

#---------------------------------------------------------------------------------
# Print GPU stats to output file at job completion

dcgmi stats --verbose --job ${SLURM_JOB_ID}

Typing sbatch submit.sh at the command line will submit the batch job using the scheduler.

Note

All sbatch jobs must be submitted from one of the front nodes pytfe01 or pytfe02. Although lines 2–27 are optional, their use is highly recommended. Omitting the SBATCH parameters will cause jobs to be scheduled with the lowest priority and will allocate limited resources to your jobs.

Submitting jobs on the Hopper partition

The hopper partitions (i.e. standard_hopper, interactive_hopper, and long_hopper) provide access to NVIDIA H100 and H200 GPUs for high-performance computing workloads. Users can choose to run jobs on H100 GPUs, H200 GPUs, or let Slurm allocate the first available GPU type.

Users can request GPUs using the --gres option in three ways:

Request the first available GPU (H100 or H200):

When no specific GPU type is requested, Slurm will allocate the first available GPU in the partition.
```
sbatch --partition=standard_hopper --gres=gpu:1 my_job_script.sh
```

Request H100 GPUs explicitly:

sbatch --partition=standard_hopper --gres=gpu:h100:1 my_job_script.sh

Request H200 GPUs explicitly:

sbatch --partition=standard_hopper --gres=gpu:h200:1 my_job_script.sh

Submitting Array of Jobs

It is sometimes necessary to submit a collection of similar jobs. This can be accomplished using job arrays with the --array option. In the example below, four separate jobs will be launched by the scheduler each with a unique $SLURM_ARRAY_TASK_ID.

submit.sh

#!/bin/bash

#SBATCH --array=0-3

# Load the software module
module load python/booth/3.10

# Access unique array task ID using environment variable
echo "Array ID: $SLURM_ARRAY_TASK_ID"
srun python3 -c "import os; x=os.environ['SLURM_ARRAY_TASK_ID']; print(x)"

Managing Jobs

The Slurm job scheduler provides several command-line tools for checking on the status of your jobs and for managing them. For a complete list of Slurm commands, see the Slurm man pages. Here are a few commands that you may find particularly useful:

The most common commands can be summarized as:

srun:: Obtain a job allocation and execute an application
sbatch:: Submit a batch script to the slurm scheduler.
sacct:: retrieve job history and information about past jobs
scancel:: cancel jobs you have submitted
squeue:: find out the status of queued jobs
sinfo:: view information about Slurm nodes and partitions.

Checking your jobs

Use the squeue command to check on the status of your jobs, and other jobs running on Pythia. The simplest invocation lists all jobs that are currently running or waiting in the job queue (“pending”), along with details about each job such as the job id and the number of nodes requested:

$ squeue

JOBID        PARTITION      NAME        USER    ST      TIME  NODES NODELIST(REASON)
standard_hopper   probe_A    vargaslo     R      0:35      1 pgpu005
standard_hopper   probe_B    vargaslo     R      0:34      1 pgpu006
standard_hopper    survey  fratnasamy     R      0:32      1 pgpu011
long_hopper       probe_C    vargaslo     R      0:30      1 pgpu012
long_hopper       survey2      hchyde    PD      0:00      1 (Resources)

Any job with 0:00 under the TIME column is a job that is still waiting in the queue. In the above case, there are not enough resources to run all jobs, so job 928 is waiting in the queue. (Resources) means that the job is waiting in the queue for available resources to free up. Another common reason for a job to be waiting in the queue is if other jobs have higher priority. This will be listed as (Priority).

To view only the jobs that you have submitted, use the --user flag

$ squeue --user=fratnasamy

JOBID        PARTITION     NAME         USER    ST      TIME  NODES NODELIST(REASON)
  926  standard_hopper   survey   fratnasamy     R      0:32      1 pgpu011
  928  standard_hopper  survey2       hchyde    PD      0:00      1 (Resources)

This command has many other useful options for querying the status of the queue and getting information about individual jobs. For example, to get information about all jobs that are waiting to run on the standard partition, enter:

$ squeue --state=PENDING --partition=standard_hopper

Alternatively, to get information about all your jobs that are running on the standard partition, type:

$ squeue --state=RUNNING --partition=standard_hopper

The last column of the output tells us which nodes are allocated for each job. For more information, consult the command-line help by typing squeue --help, or visit the official online documentation.

Canceling your jobs

To cancel a job you have submitted, use the scancel command. This requires you to specify the id of the job you wish to cancel. For example, to cancel a job with id 8885128, do the following:

$ scancel 8885128

If you are unsure what is the id of the job you would like to cancel, see the JOBID column from running squeue --user=<username>.

To cancel all jobs you have submitted that are either running or waiting in the queue, enter the following:

$ scancel --user=<username>

Viewing Past Jobs

Jobs that have completed can be viewed using sacct. This can be useful for example to see how much memory a job consumed.

# view job with jobid 993
$ sacct -j 993

# view amount of memory used during job execution
$ sacct -j <jobID> --format=User,MaxRss,MaxVMSize,Jobname,partition,start,end

# view all jobs started after a specific date for a specific user
$ sacct --starttime=2018-06-10 --user=<BoothID>