Best Practices

Auto-Requeue Python Job on SLURM Timeout

Oftentimes, jobs need more runtime than the maximum walltime limit allowed by any of the partitions. A practical solution is to configure SLURM to send a signal (such as SIGUSR1) shortly before a job times out. This gives the job a chance to gracefully prepare for termination.

Your Python script can catch this signal and then requeue itself using the scontrol requeue command. This allows the job to resume from its last saved checkpoint. Note that proper checkpointing is essential for this approach to work correctly.

SLURM Script Example

The following SLURM batch script:

  • Sets a time limit

  • Instructs SLURM to send a SIGUSR1 signal 300 seconds before timeout

#!/bin/bash
#SBATCH --job-name=auto_requeue_example
#SBATCH --time=0-01:00:00
#SBATCH --signal=USR1@300
#SBATCH --requeue
#SBATCH --open-mode=append

srun python3 my_requeue_job.py

Python Script Example

The following Python script handles the SIGUSR1 signal and triggers a requeue via scontrol requeue.

import os
import signal
import sys
import subprocess
import shlex

def handle_sigusr1(signum, frame):
    print("Caught SIGUSR1. Attempting to requeue...", flush=True)
    job_id = os.environ['SLURM_JOB_ID']

    try:
        cmd = f"scontrol requeue {job_id}"
        subprocess.run(shlex.split(cmd), check=True)
        print(f"Job {job_id} requeued successfully.")

    except subprocess.CalledProcessError as e:
        print(f"Failed to requeue job {job_id}: {e}")


# Register signal handler
signal.signal(signal.SIGTERM, handle_sigusr1)

# Simulate long-running job
import time
for i in range(3600):  # 1 hour
    print(f"Working... {i}")
    time.sleep(1)

Best Practices

  • Ensure --requeue is set in the SLURM script.

  • Handle cleanup or checkpointing in the SIGUSR1 handler if needed.

  • Monitor requeued jobs to avoid infinite loops or repeated failures.

Multicore allocation and node overload

Sometimes, multicore jobs can overload the nodes when using multicore functions. When that happens, the scheduler automatically drains the node and removes it from the pool of resources. This affects other users because they are no longer able to use the node(s). You can check the state of a node with the command:

$ sinfo -s

In order to make sure proper use of multicore allocations, we encourage users to pass on the value of the environment variable ‘SLURM_CPUS_PER_TASK’ inside their multicore functions. Most of the time, these functions use a random value different than ‘SLURM_CPUS_PER_TASK’ if the argument is not passed inside the function.

#in Python
num_cores = int(os.environ['SLURM_CPUS_PER_TASK'])
my_multicore_function(num_cores,.......)

Warning

IT reserves the right to cancel any jobs that are overloading any compute node and restrict user to 1 running job until code is successfully debugged.

Prefetch data to GPFS

Pythia provides access to fast GPFS storage via the /project_gpfs directory.

To avoid data read bottlenecks when working with large datasets users are encouraged to:

  • Prefetch data from shared project directories on NFS (e.g., /project) into the GPFS space (/project_gpfs).

This approach can significantly improve I/O performance and reduces slowdowns caused by reading directly from NFS shared project folders. Prefetching the dataset in advance prevents unnecessary GPU allocation during the data transfer.

As a first step, create a bash script that syncs or copies files from NFS to GPFS.

prefetch.sh
 1#!/bin/bash
 2#SBATCH --job-name=stage_data
 3#SBATCH --time=0-12:00:00
 4#SBATCH --partition=standard_l40s
 5
 6echo "Staging data to GPFS..."
 7
 8# source directory
 9NFS_DIR=/nfs/datasets/mydata  # NFS
10#CLOUD_URI="s3://mybucket/datasets/mydata"  # cloud
11
12# destination directory
13GPFS_DIR=/project_gpfs/${USER}/mydata
14mkdir -p "${GPFS_DIR}"
15
16# sync command (NFS or cloud)
17rsync -a --progress "${NFS_DIR}/" "${GPFS_DIR}/"  # NFS
18#aws s3 sync "${CLOUD_URI}" "${GPFS_DIR}" --only-show-errors  # AWS S3
19#rclone copy "${CLOUD_URI}" "${GPFS_DIR}" --progress  # rclone
20
21echo "Staging complete."
train_model.sh
 1#!/bin/bash
 2#SBATCH --job-name=train_model
 3#SBATCH --time=1-00:00:00
 4#SBATCH --partition=standard_l40s
 5#SBATCH --gres=gpu:1
 6
 7# Load your training environment here if needed
 8#module load python/booth/3.12
 9#module load cuda/12.8
10#source activate myenv
11
12echo "Starting training on GPFS-staged data..."
13
14python train.py --data /project_gpfs/${USER}/mydata

The submit script below uses the dependency option to hold the second job until the first job completes successfully.

submit.sh
1#!/bin/bash
2
3# submit first job and catch its job id
4jobid=$(sbatch prefetch.sh | awk '{print $4}')
5
6# submit second job to run only after first job completes successfully
7sbatch --dependency=afterok:${jobid} train_model.sh