13 Control flow#
What this notebook is about#
Your analyze.sh from Notebook 12 is a real tool, but it still does one thing,
once. The last step in Part III is to let a script decide (do this, not that)
and repeat (do it to all of them). This is control flow. At first it looks like
a pile of unrelated keywords: if, &&, while, case, for. It is not. Every
one of them is a variation on a single idea.
That idea is the exit code. Every command, when it finishes, leaves behind a
number: 0 for success, anything else for failure. All of control flow is built
on testing that number. if tests an exit code; && and || chain on it; while
loops on it; and [[ … ]] is simply a command that produces one. See the exit
code clearly and the rest stops being a list to memorise; it becomes one idea told
a few different ways.
This notebook closes Part III: by the end you can write, edit, and run a tool that sweeps a whole directory and validates its input. (As always, the data is a playground, no physics required.)
B. &&, ||, and ; — chaining on success#
The simplest control flow reads exit codes directly. Put two commands together with
&& and the second runs only if the first succeeded; with ||, only if
the first failed; with ;, always (just sequence them):
mkdir -p scratch/demo && echo "directory made, so this ran"
directory made, so this ran
The && is the everyday “do this, then that, but only if this worked”, like
mkdir d && cd d, which refuses to cd into a directory that was not created. Its
mirror, ||, is the “or else” of error handling:
false || echo "the first command failed, so the fallback ran"
the first command failed, so the fallback ran
| cmd1 && cmd2 | run cmd2 only if cmd1 succeeded (exit 0) |
| cmd1 || cmd2 | run cmd2 only if cmd1 failed (non-zero) |
| cmd1 ; cmd2 | just sequence: run cmd2 regardless |
| cmd1 | cmd2 | a pipe (Notebook 4): passes data, not about exit codes |
C. if and tests#
When the choice needs more than one line, you reach for if. And here is the
thing to hold onto: if does not test a “condition” in the algebra sense. if
runs a command and branches on its exit code. if grep -q …; then runs grep
and takes the then branch when grep succeeded. The shape:
if cmd; then
… # ran if cmd succeeded
elif cmd2; then
… # else, if cmd2 succeeded
else
… # otherwise
fi
So where do the familiar comparisons (-gt, =, -f) come in? They are not
special syntax: [[ … ]] is itself just a command whose whole job is to test
something and return 0 or 1. Watch a numeric test branch:
frames=2520
if [[ "$frames" -gt 1000 ]]; then echo "big run ($frames frames)"; else echo "small run"; fi
big run (2520 frames)
And a file test (the workhorse of input checking) using the data tree:
if [[ -f data/logs/gr2hno3-nvt.log ]]; then echo "the log is there"; fi
the log is there
The operators come in three families, curated to the ones you actually reach for:
[[ … ]] — a command that returns 0 (true) or 1 (false). Pick the family that matches your data.
| NUMERIC | -eq -ne -lt -le -gt -ge: equal, not-equal, <, ≤, >, ≥ |
| STRING | = equal, != not-equal, -z empty, -n non-empty |
| FILE | -f a file, -d a directory, -e exists, -r/-w/-x readable/writable/executable |
| LOGIC | && and, || or, ! not |
=~ | matches an extended regex (Notebook 6): [[ … ]] only |
Two notations exist, and the difference matters:
[[ … ]]is bash’s own, and the one to prefer: it does not word-split unquoted variables (Notebook 10), and it understands&&,||, and=~regex inside.[ … ](a synonym for thetestcommand) is the older, POSIX-portable form. It works everywhere, but it is fussier: you must quote your variables in it.
⚠ Inside a test, the spaces are not optional
[[ and ]] (and [ ]) are commands, so they need spaces around them and
around every operator: [[ "$n" -gt 1000 ]], never [["$n"-gt1000]]. And mind the
two equals: = compares strings, -eq compares numbers: [[ "1+1" = "2" ]] is false (different text) while [[ "1+1" -eq "2" ]] is true (same value).
In the portable [ … ], an unquoted empty variable can even break the test’s
syntax, which is exactly why [[ … ]] is the safer default in bash.
The immediate payoff is input validation, the habit that makes a script robust, and it completes the thought from Notebook 12. A script should check what it was given before it leans on it: if a required argument is missing, say so and stop.
cat > scratch/need-arg.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
if [[ $# -lt 1 ]]; then
echo "usage: need-arg.sh FILE" >&2
exit 1
fi
echo "working on: $1"
EOF
chmod +x scratch/need-arg.sh
(cd scratch && ./need-arg.sh) 2>&1 || echo "(no argument — it printed usage and exited non-zero)"
(cd scratch && ./need-arg.sh lj38.xyz)
usage: need-arg.sh FILE
(no argument — it printed usage and exited non-zero)
working on: lj38.xyz
No argument, it refuses with a usage line and a non-zero exit; one argument, it
proceeds. That if [[ $# -lt 1 ]]; then … exit 1; fi is the single most common
opening of a real-world script.
D. case — pattern dispatch#
When you are branching the same value against many possibilities, a tall
if/elif/elif/… chain gets unreadable. case is the clean tool for it: give it
a value and a list of glob patterns (Notebook 5, not regex), and it runs the
first arm that matches.
for name in run.log traj.xyz notes.txt; do
case "$name" in
*.log) echo "$name -> a log file" ;;
*.xyz) echo "$name -> a trajectory" ;;
*) echo "$name -> something else" ;;
esac
done
run.log -> a log file
traj.xyz -> a trajectory
notes.txt -> something else
Each arm ends with ;;, alternatives are written a|b), and a final *) is the
catch-all default (put it last).
| case "$x" in | match $x against the arms below |
| pat) … ;; | run this arm if $x matches the glob pat; end every arm with ;; |
| a|b) … ;; | alternatives: match a OR b |
| *) … ;; | the catch-all default: put it last |
| esac | close the block (case spelled backwards) |
Now you have everything to read the piece of
Notebook 12 we deferred: the getopts skeleton. It was a while loop wrapped
around a case, and it is just the two tools you now know:
while getopts "vn:" opt; do # loop: getopts succeeds while there are flags to read
case "$opt" in # dispatch on which flag we got
v) verbose=1 ;;
n) count="$OPTARG" ;;
*) echo "usage: …" >&2; exit 1 ;;
esac
done
That is the whole trick: getopts hands one flag per turn to case, the while
keeps going until the flags run out. Nothing magic: a loop and a switch.
E. Loops — doing it to all of them#
This is the payoff the whole course has been building toward: the recurring “do X
to all my files” goal. The everyday loop is for, and the right thing to loop
over is a glob (Notebook 5): the shell expands it to the matching files, spaces
and all:
for f in scratch/*.xyz; do
echo "$f has $(wc -l < "$f") lines"
done
scratch/run-01.xyz has 3 lines
scratch/run-02.xyz has 2 lines
The loop body ran once per file, the glob kept each name whole. for also walks
a literal list, a brace sequence {1..10}, or a C-style counter; while and
until loop on an exit code; and the while read idiom reads a file line by
line:
break leaves the loop; continue skips to the next turn.
| for f in *.xyz; do … done | iterate a glob, a list, or {1..10} |
| for ((i=0; i<n; i++)); do … done | a C-style numeric counter |
| while COND; do … done | loop while an exit code stays 0 |
| until COND; do … done | loop until an exit code becomes 0 |
| while IFS= read -r line do … done < file | read a file/stream line by line, safely |
That last one earns its odd shape. To read a file a line at a time, the safe, copyable form is exactly:
while IFS= read -r line; do
echo "saw: $line"
done < scratch/params.txt
saw: cutoff 300
saw: method PBE
The IFS= stops leading/trailing spaces from being trimmed and -r stops
backslashes from being mangled; together they read each line verbatim. (Use
while read when the logic is per line; when you want columns of data, that
is still awk’s job, Notebook 8.)
One anti-pattern deserves a hard stop, because it is everywhere and it is the Notebook-10 word-splitting bug in disguise:
⚠ Loop over a glob, never over the output of ls
You will see for f in $(ls) in the wild. Do not write it. The $(ls) is
word-split on spaces, so a file named my run.xyz is torn into my and run.xyz
and your loop runs on names that do not exist. Loop over the glob instead
(for f in *.xyz), which the shell expands to real, whole filenames. Here is the bug
and its fix, side by side.
for f in $(ls scratch); do echo "[$f]"; done
[my]
[run.xyz]
for f in scratch/*; do echo "[$f]"; done
[scratch/my run.xyz]
The first tore the name in two; the glob kept it intact. Loop over globs.
F. seq, time, and a first look at speedup#
Two small commands round out the toolkit. seq prints a sequence of numbers:
for counting and for driving sweeps when the bounds live in variables (where a brace
{1..10} cannot reach):
| N | print 1 up to N (seq 5 → 1 2 3 4 5) |
| FIRST LAST | count from FIRST to LAST (seq 2 6 → 2 3 4 5 6) |
| FIRST STEP LAST | count in steps of STEP (seq 0 2 10 → the even numbers) |
| -w | equal width: zero-pad so the numbers line up (seq -w 8 10 → 08 09 10) |
man seq{1..5} needs no external command (Notebook 5) — reach for seq when a bound or the step lives in a VARIABLE, which braces cannot expandseq 0 2 8
0
2
4
6
8
And time measures how long something takes: prefix any command with it and it
reports the wall-clock real time afterwards:
| time CMD | run CMD, then report its timing — real is the wall-clock time you actually waited for |
| time { …; } / time pipe | line | as a shell keyword it can also time a whole pipeline, group, or loop, not just one program |
help time (it is a bash keyword); the external /usr/bin/time -v additionally reports peak memory and morereal is wall-clock (what you feel); user+sys is total CPU summed across ALL cores — so for parallel work user can EXCEED real. The numbers vary run to run, so reason about ratios, not absolutestime sleep 0.2
real 0m0.203s
user 0m0.000s
sys 0m0.002s
Put loops, seq, and time together and you can ask a question that matters the
moment you reach a cluster: does throwing more workers at a job actually make it
faster? Here we run four 0.2-second tasks at one worker, then two, using the
xargs -P parallelism alluded to back in Notebook 5, now something you can read:
for w in 1 2; do
start=$(date +%s.%N)
seq 4 | xargs -P"$w" -I{} sleep 0.2
end=$(date +%s.%N)
awk -v a="$start" -v b="$end" -v w="$w" 'BEGIN { printf "%d worker(s): %.2fs\n", w, b - a }'
done
1 worker(s): 0.81s
2 worker(s): 0.41s
Two workers ran the four tasks in roughly half the wall-clock time of one: a speedup of about 2×. That is the whole game of parallel computing in miniature, and also its catch: the speedup never keeps doubling forever. Notebook 17 takes this exact experiment to real scaling data and the question of how many cores are actually worth asking for.
Exercises#
Control flow is best learned by running it. Everything below works in a fresh
scratch/; data/ stays read-only. (Scripts are written here with here-documents
so the page is reproducible; in your terminal you would type them into Vim.)
Warm-up 1 (worked) — Exit codes and chaining#
Inspect $? after a success and a failure, then use && and || to act on the
outcome.
after true: 0
after false: 1
the log exists
that one does not
✓ true returned 0 (success) and false returned non-zero (failure)
Warm-up 2 (your turn) — A first if#
Write an if/else that branches on a file test: report whether
data/logs/gr2hno3-nvt.log exists, and whether a made-up name does not.
log: present
nope: missing
✓ the file test took the correct branch for the file that exists
Applied 1 (your turn) — Validate input#
Write a script that requires one argument. If $# is zero, print a usage line to
standard error and exit 1; otherwise echo the argument. Run it both ways.
usage: guard.sh FILE
(refused: no argument)
processing: run.xyz
✓ it exits 1 with no argument and proceeds when given one
Applied 2 (your turn) — Loop over files#
In a scratch directory of .xyz files (one with a space in its name), loop with
for f in *.xyz and print each name with its line count. Do not use
for f in $(ls).
run 03.xyz: 1 lines
run-01.xyz: 3 lines
run-02.xyz: 2 lines
✓ the glob loop handled all three files, the spaced name kept whole
Applied 3 (worked) — while read + case#
Read a small key/value file line by line and dispatch on the key with case.
energy cutoff -> cutoff 300
functional -> method PBE
other -> charge 0
✓ each line was read and dispatched to the right case arm (cutoff, method, other)
Composite — putting it together (capstone: sweep a directory)#
The climax of Part III, and the goal the whole course pointed at. Extend the idea of
analyze.sh into sweep.sh: it takes a directory, validates that argument with
if, then loops over every .log in it, averaging each file’s energies with a
function (grep/awk, Notebooks 6 and 8), and prints a printf table of
file → mean. One command, a whole directory analysed.
run-01.log -101.0000
run-02.log -122.0000
run-03.log -140.5000
✓ sweep.sh validated its argument and tabulated the mean of all three logs
Optional stretch (your turn) — Speedup table#
Build the speedup teaser into a small table: run a batch of short tasks at 1, 2,
and 4 workers (a for loop over the worker counts, xargs -P"$w"), timing each
with date, and print a row per worker count. Watch the wall time fall, then start
to level off. (Exact times vary; the shape is the point. Notebook 17 develops it.)
1 worker(s): 1.62s
2 worker(s): 0.81s
4 worker(s): 0.41s
✓ the table has a timed row for each of the 1, 2, and 4 worker counts
Outlook#
Your scripts can now decide and repeat, and with that, Part III is complete. You can write a file (Notebook 9), quote it correctly (10), make it runnable (11), give it inputs and structure (12), and now have it branch, validate, and sweep an entire directory (13). That is a real command-line tool, built from nothing.
Part IV takes the tool to where the real work happens: the cluster. The environment your scripts run in and how to shape it (Notebook 14); getting onto a remote machine and moving data to and from it (15); handing your job to a scheduler that runs it for you (16); and, picking up today’s speedup teaser, measuring real scaling and justifying how many cores to actually ask for (17).
See the full Compendium Scriptorum for every command met so far, and where to find it again.