14 Environment and modules#
What this notebook is about#
Part IV is about taking everything you have built (scripts that read, write, decide, and repeat) to the place the real work happens: a shared cluster. And the very first thing that goes wrong there is almost never your science. It is the environment.
Your shell does not run in a vacuum. It runs inside a set of variables (and, on a cluster, loaded modules) that every command and every script you launch inherits. Getting that environment right is what makes the correct software available; getting it subtly wrong is the single most common cluster bug, the one behind the classic cry: “but it worked when I ran it by hand!”
This notebook is the map of that environment: shell variables versus exported ones
(§A), $PATH (§B), the startup files that set it all up, and the login-versus-batch
trap that breaks jobs (§C), the source-versus-run distinction that makes it all
work (§D), alias (§E), and finally the module system clusters use to hand you
software (§F). (As ever: no physics: we load a stand-in “code” without caring
what it computes.)
A. Variables versus the environment#
In Notebook 12 you set variables: name=value. Such a variable is private to the
current shell. The moment you run a script, you start a child shell, and the
child does not inherit your private variables. It inherits only the ones you
have exported into the environment. Here is that fact, felt. We have a tiny
child script that just reports one variable:
Set the variable the ordinary way and run the child: it sees nothing, because a plain variable does not cross into the child:
GREETING="hello from the parent"; bash scratch/child.sh
the child sees GREETING as: [(unset)]
Now export it (promoting it from a shell variable to an environment
variable) and run the very same child again:
export GREETING; bash scratch/child.sh
the child sees GREETING as: [hello from the parent]
This time it crosses. That is the whole mental model of the environment: a child inherits exactly the exported variables, and nothing else. Hold onto it: almost every “my job can’t see X” problem is this picture with X unexported.
| export NAME=value | set a variable AND mark it for inheritance, in one step |
| export NAME | mark an already-set variable for inheritance |
| -p | list every exported variable currently in the environment |
help export; see the whole environment with printenv or envThe flip side of export is seeing what is in the environment, with printenv
(or the env command), and note that our newly-exported GREETING is now in it,
while a plain shell variable would not be:
| printenv | list the whole environment, one NAME=value per line |
| printenv NAME | print just one variable's value (empty, non-zero exit, if it is unset) |
man printenv; the env command with no arguments does the same listingexport) does not appear, which is exactly the environment-vs-shell-variable distinction made visibleprintenv GREETING
hello from the parent
B. $PATH revisited#
You already met the most important environment variable in Notebook 11: $PATH,
the colon-separated list of directories the shell searches for a command you type by
name. It is just an environment variable, so you shape it with export. The
canonical move, the one that finally delivers Notebook 11’s “promote your script to
a command,” is to prepend your own bin directory:
export PATH="$HOME/bin:$PATH"
Prepending (your directory first) means your version wins when names collide; the
existing $PATH is kept on the end so every normal command still works. Watch it
make a script callable by bare name:
export PATH="$ROOT/scratch/bin:$PATH"; greet-tool
greet-tool ran, found by name on PATH
The bare name resolved (no ./) because its directory is now on $PATH. One
rule guards this:
⚠ Extend PATH, never replace it
Always write export PATH="newdir:$PATH", keeping the old $PATH on the end.
If you write a bare export PATH="newdir" you have clobbered it: the shell can
no longer find ls, grep, or anything else, and your session is effectively
broken until you fix it. The :$PATH on the end is not optional decoration; it is
the rest of your system.
C. Startup files: login versus non-login#
Where do all these exports and PATH tweaks usually live? In a startup file that the shell reads automatically. But which file, and whether it is read at all, depends on the kind of shell, and this is the linchpin of the whole notebook.
| ~/.bash_profile · ~/.profile | login shells: when you ssh in or log in at a console |
| ~/.bashrc | non-login, interactive shells: opening a new terminal tab |
| (none) | non-interactive shells (a script, or a batch job) read no rc file |
The common convention is that ~/.bash_profile simply sources ~/.bashrc, so a
login shell ends up with your interactive setup too. But look at that last row,
because it is the trap. A batch job runs your script in a fresh non-interactive
shell — so let us start exactly that kind of shell and ask it:
bash -c 'case $- in
*i*) echo "this shell is INTERACTIVE — it reads ~/.bashrc" ;;
*) echo "this shell is NON-interactive — like a script or a batch job, it reads NO rc file" ;;
esac'
this shell is NON-interactive — like a script or a batch job, it reads NO rc file
⚠ Why your cluster job can’t find your modules
A scheduler runs your job script in a non-login, non-interactive shell. So it
never sources your ~/.bashrc: none of your aliases, your $PATH edits, or
your module load lines from there exist inside the job. This is the reason for
“it ran when I typed it, but the batch job says command not found.” The fix is not
to fight the shell: it is to put the setup your job needs inside the job script
itself (Notebook 16), so it does not depend on being inherited.
D. source versus execute — the key distinction#
That fix rests on one idea, and it is the most important tool in this notebook. When
you run ./script (Notebook 11), it runs in a child shell, so anything it
changes (variables, $PATH, loaded modules) vanishes the instant it finishes. When
you source it instead, its lines run in your current shell, so its changes
stay. We have a script that sets a variable:
Run it as a child with ./ (here, bash …) and the variable does not survive:
the child set it, then exited and took it with it:
bash scratch/setvar.sh; echo "after running it: [${MSG:-(unset)}]"
after running it: [(unset)]
Now source the same script: its line runs here, so the variable persists:
source scratch/setvar.sh; echo "after sourcing it: [$MSG]"
after sourcing it: [set inside the script]
| source FILE / . FILE | execute FILE's lines in this shell — . is the POSIX spelling of the same thing |
help source./script (Notebook 11): ./script runs in a CHILD shell and its changes vanish when it exits; source runs the lines HERE, so its changes stay. It is how .bashrc and module load take effectThat is exactly why you source ~/.bashrc to apply changes without opening a new
terminal, and why module load (below) has to alter your current shell to work
at all. Run vs. source is the difference between “did something and left” and
“changed where I’m standing.”
E. alias — interactive shorthand#
A small convenience for the prompt: an alias gives a short name to a longer
command. The classic lives in everyone’s .bashrc:
| alias ll='ls -lh' | define an alias (keep your favourites in .bashrc) |
| alias | list every alias currently defined |
| unalias NAME | remove an alias |
help aliasalias ll='ls -lh'; alias ll
alias ll='ls -lh'
At an interactive prompt, typing ll now expands to ls -lh. But there is a catch
worth stating plainly, and it ties straight back to Notebook 12:
bash scratch/tryalias.sh
the child script does NOT see the ll alias
A script cannot see your aliases: they are an interactive-only convenience. So when a script needs a reusable shorthand, you do not reach for an alias; you write a function (Notebook 12), which a script defines for itself.
F. The module system#
On a cluster you do not install software, and it is not all sitting on $PATH
waiting for you. There are dozens of programs, often several versions of each
with conflicting dependencies, and you opt into exactly the ones you want with the
module system. It is the cluster embodiment of everything in this notebook:
module load works precisely by modifying your current environment (adding to
$PATH, setting variables) the way a sourced script does.
| module avail | list the software (and versions) available to load |
| module load NAME/VER | add a module to your environment (puts it on PATH, sets its variables) |
| module list | show what you currently have loaded |
| module purge | unload everything — back to a clean environment |
| module swap A B / module unload NAME | replace one module with another / drop a single one |
module help; on Euler and most clusters this is the Lmod or environment-modules systemmodule modifies the CURRENT environment (like source, not ./), so its effect is session-local — which is why a batch job must module load what it needs INSIDE the job script (Notebook 16), not lean on your login setupStart by asking what is available. (We have set up a demo “simulation code” in two
versions, plus a second tool family; on a real cluster the list runs to hundreds.)
module avail writes its listing to standard error, which is normal:
module avail 2>&1
------------ /home/runner/work/bash-primer/bash-primer/modulefiles -------------
analysis-tools/3.2 democode/1.0 democode/2.0
Key:
modulepath
Before loading anything, the democode command does not exist: it is not on
$PATH:
command -v democode || echo "democode is not available yet"
democode is not available yet
Now load version 1.0. With no fanfare, it puts democode on your $PATH and
sets a variable, and suddenly the command runs:
module load democode/1.0; democode
democode 1.0 — stand-in simulation code (loaded via the module system)
echo "the module set DEMOCODE_VERSION=$DEMOCODE_VERSION"
the module set DEMOCODE_VERSION=1.0
module list shows what you have loaded, and module purge clears
everything, taking democode straight back off your $PATH:
module list 2>&1
Currently Loaded Modulefiles:
1) democode/1.0
module purge; command -v democode || echo "after purge, democode is gone again"
after purge, democode is gone again
That is the whole loop (avail, load, list, purge, plus swap to change
versions), and the reason it matters here: because a module only changes the
current environment, a batch job must run these module load lines inside
itself. That is exactly where Notebook 16 picks up.
Exercises#
Everything below is session-local and reversible: scratch files, a scratch bin/,
and module loads we purge afterwards. data/ stays read-only. (Scripts are written
with here-documents for reproducibility; in your terminal you would use Vim.)
Warm-up 1 (worked) — Inheritance, felt#
Set a variable, run a child script that reads it: first unexported (the child sees nothing), then exported (the child sees it).
unexported:
[(unset)]
exported:
[lj38]
✓ the child saw nothing until the variable was exported
Warm-up 2 (your turn) — Extend $PATH#
Put a script in a scratch bin/, prepend that directory to $PATH, and call the
script by its bare name (no ./).
mytool reporting in
✓ the script ran by bare name once its directory was on PATH
Applied 1 (your turn) — source versus ./#
A script sets a variable. Run it with ./ (a child, where the variable does not persist)
and then source it (it does). Report the variable after each.
after ./: [(unset)]
after source: [/opt/democode]
✓ the variable persisted only after source, not after running as a child
Applied 2 (your turn) — Load software with module#
List what is available, load democode/1.0, confirm it is on $PATH and listed,
then purge and confirm it is gone.
------------ /home/runner/work/bash-primer/bash-primer/modulefiles -------------
analysis-tools/3.2 democode/1.0 democode/2.0
Key:
modulepath
loaded; democode is at: /home/runner/work/bash-primer/bash-primer/opt/democode-1.0/bin/democode
Currently Loaded Modulefiles:
1) democode/1.0
after purge: none
✓ load put democode on PATH and set its version; purge removed it
Composite — putting it together (prepare an environment)#
The capstone, and the shape of every cluster setup you will ever write. Compose one
small env-setup file that does three things (exports a variable, extends
$PATH with your scratch bin/, and module loads the demo code), then source
it and confirm that both your own tool and the loaded code now run. (In Notebook
16 this exact block moves inside a job script.)
analyze: processing lj38 with democode
democode 1.0 — stand-in simulation code (loaded via the module system)
✓ sourcing the env file exported the variable, extended PATH, and loaded the code — all three tools then ran
Optional stretch (your turn) — A .bashrc-style file#
Build a scratch rc file with an export, a function, and an alias;
source it; and confirm the export and the function work, while a child script
cannot see the alias (so a script would use the function instead).
EDITOR is now: vim
hello from a function
alias ll='ls -lh'
child: no ll alias
✓ source applied the export and the function; the child script could not see the alias
Outlook#
You can now read and shape the environment your scripts run in (exported variables,
$PATH, startup files, source versus run, and the module system), and, crucially,
you know why a batch job may inherit none of it. That last point is the bridge to
the rest of Part IV. Next (Notebook 15): actually getting onto a remote machine and
moving your scripts and data there: ssh, scp, rsync, archiving with tar, and
keeping a long job alive after you log out.
export— promote a shell variable to an ENVIRONMENT variable, so every child process and script inherits it.printenv— print the environment — the exported variables the shell passes down to its children.source— run a script in the CURRENT shell (not a child), so the variables, functions, and PATH changes it makes persist.alias— give a short name to a longer command — typed shorthand for interactive use.module— load and unload software on a cluster — adding a chosen version of a tool to your environment on demand.
See the full Compendium Scriptorum for every command met so far, and where to find it again.