16 SLURM submission scripts#
What this notebook is about#
Everything in Notebooks 14 and 15 got you onto the cluster and your data across to it. This one is about the thing you actually came for: running the work, and on a shared machine you do not just run it, you hand it to a scheduler.
A submission script is the whole idea, and it is nothing new: it is a bash script
(Part III) with a short header of #SBATCH directives that tell the scheduler
what resources you want. You write it, submit it, and check on it; it runs
later, on some compute node, in a fresh non-login shell, which is exactly why
the environment lesson from Notebook 14 is about to pay off. The loop is:
submit → queue → run → collect.
A real scheduler we can stand in for
There is no SLURM scheduler inside this page, so the course ships a thin mock:
sbatch runs your job’s body right here (in the background, in a fresh shell) and
writes a real slurm-<jobid>.out; squeue, scancel, sacct, and sinfo all
behave plausibly. Because #SBATCH lines are ordinary comments to bash, the
scripts you write here are exactly what you would submit on ETH’s Euler or
any other SLURM cluster.
The one thing the mock cannot reproduce is the queue wait: here your job starts immediately, where on a shared cluster it waits its turn behind hundreds of others. That wait is the scheduler’s whole reason to exist, so picture the line even as we skip it. (As ever: no physics: we load a stand-in “code” and never ask what it computes.)
A. Why a scheduler#
A cluster is shared. Hundreds of people want to run hours-long jobs on the same few thousand cores, so you cannot simply log in and launch your work: you would trample everyone else (and they, you). Instead the machine runs a scheduler (SLURM, on Euler and most academic clusters). You describe the job and the resources it needs; SLURM puts it in a queue, and when the resources are free it runs your job on a compute node and saves the output for you to collect.
The machine is carved into partitions (named queues, often by time limit).
sinfo shows them: the menu you are choosing from:
| sinfo | list the partitions with their STATE (idle, mix, alloc) and TIMELIMIT |
man sinfo; the partition names feed your #SBATCH --partition= choicenormal.4h). Check sinfo and your cluster's docs before guessing a --partitionsinfo
PARTITION AVAIL TIMELIMIT NODES STATE
normal.4h up 4:00:00 48 idle
normal.24h up 24:00:00 96 mix
normal.120h up 120:00:00 24 alloc
(Real Euler partitions are time-based, normal.4h, normal.24h, and so on, and
busier; this is the mock’s tidy stand-in.) One rule of etiquette follows immediately
from “shared”: never run heavy work on the login node. The login node is the
shared front desk where everyone types; you submit real computation to a compute
node, you do not run it where you land.
B. Anatomy of a submission script#
A submission script has three parts, in order: a shebang, a header of
#SBATCH directives, then the body (the actual commands). Here is the
smallest complete one: write it and look at its three parts:
cat > first-job.sh <<'EOF'
#!/usr/bin/env bash
#SBATCH --job-name=first
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --mem-per-cpu=1G
echo "Hello from batch job $SLURM_JOB_ID, running as '$SLURM_JOB_NAME'."
echo "I ran on a compute node, in a shell all my own."
EOF
The #SBATCH lines look like comments, and to bash they are comments, which
is why the script still runs as an ordinary file. To SLURM they are resource
requests, read before the job starts. They must come before any real command.
Here is the workhorse set:
| --job-name=NAME | a label for the job, shown in squeue |
| --time=HH:MM:SS | wall-clock limit; the job is KILLED when it is exceeded |
| --ntasks=N | number of tasks (e.g. MPI ranks); 1 for a serial job |
| --cpus-per-task=N | cores per task: for a threaded / OpenMP code |
| --nodes=N | how many compute nodes to spread across |
| --mem-per-cpu=SIZE | memory per core, e.g. 2G; too low and the job is OOM-killed |
| --partition=NAME | which queue (from sinfo) to submit to |
| --output=FILE | where output goes (default slurm-%j.out; %j = the job id) |
| --array=A-B | submit a job array: many tasks from one script (§E) |
⚠ Request realistically — too little is as bad as too much
Two directives bite hardest. If --time is shorter than the job needs, SLURM
kills it the moment the limit passes: hours of work, gone at 99%. If
--mem-per-cpu is lower than the job needs, it is OOM-killed. (And asking
for too much of either just makes you wait longer in the queue, since SLURM must
find that much free, and on many clusters it bills your allocation.) Estimate from
a real run, then add a margin.
Now submit it. sbatch hands the script to the scheduler and prints a job
ID: the number you will use to track it:
| sbatch script.sh | queue the script; it runs later on a compute node, its output going to slurm-<jobid>.out |
| --array=1-N | submit a job ARRAY — run the script N times, each task with its own $SLURM_ARRAY_TASK_ID |
| (resources) | set with #SBATCH directives at the top of the file, not as command-line flags |
man sbatch; it returns Submitted batch job <id> — note the id to track the jobmodule load what you need INSIDE the script (Notebook 14)sbatch first-job.sh
[1] 4808
Submitted batch job 1000
The job ran on a “compute node”, and (this is the part that surprises everyone the
first time) its output did not come back to your screen. It went to a file,
slurm-<jobid>.out, in the directory you submitted from. That is where results live;
go and find it:
ls slurm-*.out
slurm-1000.out
cat slurm-*.out
Hello from batch job 1000, running as 'first'.
I ran on a compute node, in a shell all my own.
There is the whole loop in miniature: you wrote a script, sbatch queued it, it ran
elsewhere, and it left its output in a file for you to collect.
C. The environment inside the job#
Here is the single most important practical lesson in this notebook, and it follows
straight from Notebook 14. Your job runs in a fresh, non-login shell on a compute
node. That shell does not run your ~/.bashrc, and it does not inherit the
modules you loaded by hand on the login node. So if your script just calls a tool,
expecting it to be on PATH the way it was when you tested by hand: it will not be.
Watch it fail. First, load our stand-in code here, in this interactive shell, and confirm it is available:
module load democode/1.0
command -v democode
/home/runner/work/bash-primer/bash-primer/opt/democode-1.0/bin/democode
It is right there on our PATH. Now submit a job that simply runs it, without
loading it inside the script:
cat > no-module.sh <<'EOF'
#!/usr/bin/env bash
#SBATCH --job-name=no-module
democode
EOF
sbatch no-module.sh
[1] 4847
Submitted batch job 1001
cat slurm-*.out
no-module.sh: line 3: democode: command not found
command not found: even though democode was loaded right here when we
submitted. The job’s fresh shell never saw it. The fix is the whole lesson: load
the environment inside the script, so the job sets itself up no matter what your
login shell happened to have:
cat > with-module.sh <<'EOF'
#!/usr/bin/env bash
#SBATCH --job-name=with-module
module load democode/1.0
democode
EOF
sbatch with-module.sh
[1] 4878
Submitted batch job 1002
cat slurm-*.out
democode 1.0 — stand-in simulation code (loaded via the module system)
Same job, one line added, and now it works. Every submission script should set up
its own environment (module load, and export what you need) in its body. It is
the difference between “it worked when I ran it by hand” and a job that actually runs.
D. Submit and monitor#
You rarely submit one job and walk away. The real rhythm is submit, then watch:
is it queued, running, done? Three commands cover it. squeue shows the queue,
scancel cancels a job, and sacct reports on jobs that have already
finished.
| -u "$USER" | only YOUR jobs (the everyday view) |
| -j JOBID | one specific job by id |
| ST column | the state: PD pending (waiting in line), R running, CG completing |
man squeue; pair with watch squeue (Notebook 15) to watch the queue update livesacct for its history and read slurm-<jobid>.out for its result| scancel JOBID | cancel that one job |
| -u "$USER" | cancel ALL of your jobs at once (use with care) |
man scancel; get the id from squeuescancel -u "$USER" kills every job you have queued, not just the one you meant| sacct | your recent jobs and how they ended (COMPLETED, FAILED, CANCELLED, TIMEOUT) |
| -j JOBID | one job by id |
man sacct; squeue is the LIVE queue, sacct is the history after the factsqueue is not lost — sacct shows how it ended: TIMEOUT means --time was too short, OUT_OF_MEMORY means --mem was too lowSubmit a longer job and watch it move through the system. First, there it is in the
queue, running (R), with the job id, name, and partition:
jobid=$(sbatch long-job.sh | awk '{print $NF}')
echo "submitted, tracking job $jobid"
submitted, tracking job 1003
squeue -u "$USER"
JOBID PARTITION NAME USER ST TIME NODES NODELIST
1003 normal long-job runner R 0:01 1 compute-01
On a real cluster you would watch squeue (Notebook 15) and wait for it to finish.
Here, suppose you spot a mistake and want it gone: scancel it by id:
scancel "$jobid"
squeue -u "$USER"
JOBID PARTITION NAME USER ST TIME NODES NODELIST
The queue is empty again. And after the fact (whether a job finished or was
cancelled) sacct is the history book that says how it ended:
sacct
JobID JobName State ExitCode
------------ ------------ ----------- --------
1003 long-job CANCELLED 0:0
CANCELLED, as expected. (A finished job would read COMPLETED; a job that ran past
its --time reads TIMEOUT; one that overran its memory, OUT_OF_MEMORY.) The
loop, then, is: sbatch → squeue to watch → read slurm-<jobid>.out for the
result, or sacct for the verdict.
E. Job arrays — one script, many tasks#
The cluster’s real power is doing the same work over a whole dataset at once. You
have 200 trajectories to analyze; you do not write 200 scripts, and you do not loop
200 sbatches. You write one script and submit it as a job array.
#SBATCH --array=1-N tells SLURM to run the script N times. Each run (each
task) is identical except for one variable, $SLURM_ARRAY_TASK_ID, which is
its index (1, 2, 3, …). You use that index to pick which item this task handles,
exactly the indexing logic of a loop (Notebook 13), but the scheduler runs the tasks
in parallel for you. Here is a sweep over a small list of trajectory files:
We have a list of files (one per line) and a script whose task picks the line matching its array index, then “analyzes” it (here: reports the atom count):
cat files.txt
traj_01.xyz
traj_02.xyz
traj_03.xyz
cat > sweep.sh <<'EOF'
#!/usr/bin/env bash
#SBATCH --job-name=sweep
#SBATCH --array=1-3
#SBATCH --time=00:10:00
module load democode/1.0 # set up the environment (§C)
file=$(sed -n "${SLURM_ARRAY_TASK_ID}p" files.txt) # this task's file
atoms=$(head -n 1 "$file") # the "analysis"
echo "task ${SLURM_ARRAY_TASK_ID}: ${file} has ${atoms} atoms"
EOF
Submit it once; SLURM expands it into three tasks:
sbatch sweep.sh
[1] 4988
Submitted batch job 1004
Each task wrote its own output file, slurm-<jobid>_<task>.out:
ls slurm-*_*.out
slurm-1004_1.out slurm-1004_2.out slurm-1004_3.out
cat slurm-*_*.out
task 1: traj_01.xyz has 12 atoms
task 2: traj_02.xyz has 38 atoms
task 3: traj_03.xyz has 7 atoms
Three files processed from one submission, each by its own task. Collecting the per-task outputs into a single result is the everyday last step:
cat slurm-*_*.out | sort > results.txt; cat results.txt
task 1: traj_01.xyz has 12 atoms
task 2: traj_02.xyz has 38 atoms
task 3: traj_03.xyz has 7 atoms
The variables SLURM sets inside the job (you read them, you never set them):
| $SLURM_JOB_ID | the job's id (the number sbatch returned) |
| $SLURM_JOB_NAME | the --job-name you gave it |
| $SLURM_SUBMIT_DIR | the directory you ran sbatch from |
| $SLURM_ARRAY_TASK_ID | which array task this is: the index you sweep on |
| $SLURM_CPUS_PER_TASK | the --cpus-per-task you requested |
Exercises#
The runnable ones submit through the mock: each lives in a fresh scratch/, every
slurm-*.out is cleaned up between them, and data/ stays read-only. Job IDs vary,
so the checks read content, not numbers. (When a hidden step needs the
background job to finish before reading its output, it waits, the mock’s stand-in
for “poll squeue until it is done”.)
Warm-up 1 (worked) — Anatomy and submit#
Write a minimal script (shebang + a couple of #SBATCH lines + an echo body),
sbatch it, find its slurm-<jobid>.out, and read it.
[1] 5044
Submitted batch job 1005
--- output file: ---
slurm-1005.out
--- its contents: ---
hello from job 1005 on the cluster
✓ the script was submitted and its output landed in slurm-<jobid>.out
Warm-up 2 (your turn) — Submit and monitor#
Submit a job, see it in squeue, then scancel it and confirm with sacct that it
is recorded as cancelled.
submitted job 1006
JOBID PARTITION NAME USER ST TIME NODES NODELIST
1006 normal monitor-me runner R 0:01 1 compute-01
--- queue after cancel: ---
JOBID PARTITION NAME USER ST TIME NODES NODELIST
--- history: ---
JobID JobName State ExitCode
------------ ------------ ----------- --------
1006 monitor-me CANCELLED 0:0
✓ the job appeared in the queue and its end-state was recorded by sacct
Applied 1 (your turn) — The environment inside the job#
Submit a script that runs democode without loading its module (it fails), then
one that module loads it inside (it works): the §C lesson, in your hands.
[1] 5159
Submitted batch job 1007
--- without module load: ---
bare.sh: line 3: democode: command not found
[1] 5190
Submitted batch job 1008
--- with module load inside: ---
democode 1.0 — stand-in simulation code (loaded via the module system)
[1] 5221
Submitted batch job 1009
[1] 5251
Submitted batch job 1010
✓ the job failed without the module load and succeeded with it loaded inside
Applied 2 (worked) — A real Euler submission script#
Write a properly-formed script with the full workhorse header (--partition,
--time, --ntasks, --cpus-per-task, --output), set up its environment, and
submit it. Then “lint” it: confirm the directives are present.
--- the #SBATCH header: ---
#SBATCH --job-name=production
#SBATCH --partition=normal.4h
#SBATCH --time=02:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=2G
#SBATCH --output=run-%j.out
[1] 5296
Submitted batch job 1011
--- output (note the custom --output name): ---
running production job 1011 on 1 core(s)
democode 1.0 — stand-in simulation code (loaded via the module system)
✓ the script has a well-formed Euler header, honoured --output, and ran
Composite — putting it together (a job-array sweep)#
The realistic “run my analysis over the whole dataset” workflow, composing the file
list (Notebook 5), array indexing (Notebook 13), the script (Notebook 12), and the
environment inside (Notebook 14). Write one array script that processes one
trajectory per task ($SLURM_ARRAY_TASK_ID indexing a file list) module loads
inside, writes each task’s result, then collect the results into a table.
[1] 5341
Submitted batch job 1012
--- per-task outputs: ---
slurm-1012_1.out slurm-1012_2.out slurm-1012_3.out slurm-1012_4.out
--- collected results table: ---
frame_01.xyz 12
frame_02.xyz 38
frame_03.xyz 54
frame_04.xyz 9
✓ the array ran four tasks, one per file, and the collected table has all four results
Optional stretch (conceptual) — Generate a script, and the resource question#
No grade. Two directions. (a) A submission script is just text, so you can generate one with a heredoc (Notebook 12), handy when a parameter changes per run:
generated:
run-1.sh run-2.sh run-4.sh
--- run-4.sh: ---
#!/usr/bin/env bash
#SBATCH --job-name=scaling-4
#SBATCH --cpus-per-task=4
#SBATCH --time=00:30:00
echo "would run on 4 core(s)"
(b) Every script in this notebook simply guessed at --ntasks and
--cpus-per-task. How many should you actually request? Asking for more cores does
not automatically run faster, and wastes your allocation while you wait longer in
the queue. That question is the whole of Notebook 17. (And for an interactive
session on a compute node (a shell to test in, rather than a batch job) the tool is
srun; read your cluster’s docs for it.)
Outlook#
You can now write a real submission script, set up its environment from the inside,
submit and monitor it with sbatch/squeue/scancel/sacct, and sweep an entire
dataset with a single job array. That is the working cluster loop, end to end.
But every script you wrote guessed at --ntasks and --cpus-per-task. Ask for
too few and the job crawls; ask for too many and you wait longer in the queue, waste
your allocation, and (the part that surprises people) often do not run any faster
at all. How many should you actually request? Next (Notebook 17): read real
scaling data, compute parallel efficiency, and justify the number: the reasoning
behind a real CSCS or LUMI allocation proposal.
sbatch— submit a batch script to the scheduler — it queues the job and prints a job ID.squeue— show the job queue — what is pending, running, and whose.scancel— cancel a queued or running job by its job ID.sacct— report accounting/history for jobs that have run — their state and exit code.sinfo— show the cluster's partitions (queues) — their names, time limits, and how busy they are.
See the full Compendium Scriptorum for every command met so far, and where to find it again.