18 make and Makefiles

18 make and Makefiles#

What this notebook is about#

Everything you have automated so far has been imperative: a script says do this, then this, then this, and it does the whole thing, start to finish, every single time. That is fine until the pipeline is long and only one input changed: re-running the entire analysis to update one result is wasteful, and remembering the right order by hand is error-prone.

make is declarative. Instead of writing the steps, you describe the dependencies: what each output is built from, and the recipe to build it. make then works out the order itself and, the whole point, rebuilds only what is out of date. Change one trajectory, type make, and only that trajectory’s analysis re-runs; everything else is left untouched.

This is the capstone. A single Makefile will tie together the threads of the entire course: the Part II extraction (grep/awk), the Part III scripts, and the Part IV sweep, composed into one make. And there is a fitting closing note: the website you are reading these notebooks on is itself built with make. You are about to learn the tool that built the course.

A. The problem `make` solves#

Picture the pipeline you have been building by hand: for each trajectory, extract its energies, compute a result, then aggregate the results into a table. As a script it is a wall of commands that runs everything, in full, on every invocation, and if you reorder two steps by mistake, it silently does the wrong thing.

make inverts this. You declare three things per output: its name (the target), what it depends on (the prerequisites), and how to build it (the recipe). make reads the whole web of dependencies, computes a correct order, and runs a recipe only if its target is older than something it depends on. Declare the what; let make decide the when.

B. Anatomy of a rule#

A Makefile is a list of rules. Each rule is a target, a colon, its prerequisites, then one or more TAB-indented recipe lines:

A rule — "to make target you need prerequisites; here is how." make runs the recipe only when the target is older than a prerequisite.

target: prereqs	the thing to build, and what it is built from
⇥ recipe	the shell command(s) that build it — indented with a real TAB

Here is the smallest useful Makefile: one rule that builds summary.txt from data.txt. The recipe must be indented with a real TAB (more on that footgun in a moment). Throughout this notebook a small helper, maketab, writes the Makefile and converts the leading spaces you see in the cell into that required tab, so the pages render cleanly; in your own editor you simply press Tab. Write it and run make:

maketab Makefile <<'EOF'
summary.txt: data.txt
    wc -l < data.txt > summary.txt
EOF
make

wc -l < data.txt > summary.txt

cat summary.txt

make saw that summary.txt did not exist, found the rule for it, and ran the recipe. Run make again and it does nothing: the target is now newer than its prerequisite, so there is nothing to rebuild:

make

make: 'summary.txt' is up to date.

That “‘summary.txt’ is up to date” is the feature. Now change the input and re-run: make notices and rebuilds:

printf 'date\n' >> data.txt
make

wc -l < data.txt > summary.txt

cat summary.txt

⚠ The TAB, not spaces — the one error everyone hits

Recipe lines must be indented with a real TAB character, never spaces. It is the single most common Makefile mistake, and the error message is famously cryptic. Here is the same rule with its recipe indented by spaces: watch it fail:

make -f Makefile.bad 2>&1 || echo "(make stopped — exactly the error to recognise)"

Makefile.bad:2: *** missing separator.  Stop.
(make stopped — exactly the error to recognise)

*** missing separator almost always means spaces where a TAB belongs. A companion subtlety: each recipe line runs in its own shell, so a cd on one line does not carry to the next: chain with &&, or keep each line self-contained.

C. Running `make`

make — run a Makefile — build each target from its prerequisites, rebuilding only what is out of date.

make	build the first (default) target in `Makefile`
make TARGET	build a specific target by name (e.g. `make clean`, `make results.txt`)
-n	dry-run — PRINT the commands it would run, without running them (the `--dry-run` habit, Notebook 15)
-j N	run independent recipes in parallel across N jobs (the `xargs -P` idea, Notebooks 5 & 17)

more: man make; the GNU manual is thorough (info make)

Watch out: recipe lines MUST be indented with a real TAB, never spaces, or make stops with *** missing separator — the single most common Makefile mistake

Plain make builds the first (default) target; make TARGET builds a named one. make -n is the dry run: it prints the commands it would run without running them, the same “look before you leap” habit as rsync --dry-run (Notebook 15):

rm -f summary.txt
make -n

wc -l < data.txt > summary.txt

Nothing was built (no summary.txt yet, check if you like); -n only showed the plan. And make -j N runs independent recipes in parallel across N jobs, the same idea as xargs -P (Notebook 5) and the speedup you measured in Notebook 17, now applied to the build graph.

D. Variables, automatic variables, and `.PHONY`#

Three pieces of Makefile vocabulary make rules concise and reusable.

Variables hold values you reuse, and $(wildcard ...) pulls in files by glob, the Notebook-5 idea in Makefile form:

FILES    = $(wildcard *.xyz)        # every .xyz, like a glob
ENERGIES = $(FILES:.xyz=.energy)    # the same names with .energy instead

Automatic variables stand for parts of the current rule, so a recipe need not repeat filenames: they are to recipes what $1 $2 "$@" (Notebook 12) are to scripts:

Automatic variables — set by make inside each recipe; you read them, never assign them.

$@	the target being built
$<	the first prerequisite
$^	all the prerequisites

.PHONY marks targets that are not files (clean, all, and the like) so make always runs them and never confuses them with a real file of the same name:

maketab Makefile <<'EOF'
OUT = summary.txt

$(OUT): data.txt
    wc -l < data.txt > $@

.PHONY: clean
clean:
    rm -f $(OUT)
EOF
make

wc -l < data.txt > summary.txt

The recipe wrote to $@ (which is summary.txt), and a variable named the output in one place. Now make clean (a phony target) removes it:

make clean
ls summary.txt 2>&1 || echo "summary.txt is gone — clean did its job"

rm -f summary.txt
ls: cannot access 'summary.txt': No such file or directory
summary.txt is gone — clean did its job

Why .PHONY matters

If a file named clean ever appeared in the directory, then without .PHONY, make clean would see that the “target” clean already exists, decide it is up to date, and do nothing. .PHONY: clean tells make “this is an action, not a file”, so it always runs. Same for all, test, install.

E. Pattern rules#

You rarely want one rule per file. A pattern rule uses % as a wildcard to say “to make any X.energy from the matching X.xyz, do this”: one rule for every trajectory. It is the Notebook-5 goal (“do X to all my files”), now incremental and declarative:

%.energy: %.xyz
    grep 'E =' $< | awk '{print $$NF}' | sort -n | head -1 > $@

Read it as: for any target ending .energy, the prerequisite is the same name ending .xyz; the recipe extracts the energies (the Notebook-6/8 idiom) and writes the smallest to $@. Write that rule once and make applies it to every file.

One detail in that recipe: awk '{print $$NF}', with a double dollar. make claims a single $ for its own variables, so to pass a literal $ through to the shell (here, awk’s $NF last-field) you double it: $$ in the Makefile becomes $ by the time the recipe runs. A single $NF would be eaten by make: a quiet cousin of the TAB gotcha.

F. The synthesis Makefile#

Here is where the whole course converges. We have a set of trajectories; for each we extract energies (grep/awk, Part II), reduce to a per-file result, then aggregate into one table, and make runs it all, in order, rebuilding only what changed. Set up three trajectories and the Makefile:

maketab Makefile <<'EOF'
FILES    = $(wildcard *.xyz)
ENERGIES = $(FILES:.xyz=.energy)

all: results.txt

# pattern rule: each trajectory -> its minimum energy (Part II extraction)
%.energy: %.xyz
    grep 'E =' $< | awk '{print $$NF}' | sort -n | head -1 > $@

# aggregate every per-file result into one labelled table
results.txt: $(ENERGIES)
    for f in $^; do printf '%s\t%s\n' "$${f%.energy}" "$$(cat $$f)"; done > $@

.PHONY: clean
clean:
    rm -f *.energy results.txt
EOF
make

grep 'E =' alpha.xyz | awk '{print $NF}' | sort -n | head -1 > alpha.energy
grep 'E =' beta.xyz | awk '{print $NF}' | sort -n | head -1 > beta.energy
grep 'E =' gamma.xyz | awk '{print $NF}' | sort -n | head -1 > gamma.energy
for f in alpha.energy beta.energy gamma.energy; do printf '%s\t%s\n' "${f%.energy}" "$(cat $f)"; done > results.txt

One make extracted all three energies and aggregated them. There is the table:

cat results.txt

alpha	-12.3
beta	-22.4
gamma	-7.7

Now the payoff. Change one trajectory and re-run: make rebuilds only that file’s energy (and the table that depends on it), leaving the other two untouched:

printf '1\nE = -30.0\nC 0 0 0\n' >> beta.xyz
make

grep 'E =' beta.xyz | awk '{print $NF}' | sort -n | head -1 > beta.energy
for f in alpha.energy beta.energy gamma.energy; do printf '%s\t%s\n' "${f%.energy}" "$(cat $f)"; done > results.txt

Only beta.energy and results.txt rebuilt; alpha and gamma were already up to date, so make skipped them. On a real analysis of hundreds of trajectories, that incremental rebuild is the difference between seconds and hours. And make -j would run the independent per-file extractions in parallel (Notebook 17). One declarative file; the entire course pipeline.

Exercises#

Every Makefile and its outputs live in a fresh scratch/; data/ stays read-only. make is timestamp-based, so the incremental demos use a controlled touch, and the checks look at which targets built or rebuilt — not at exact times.

Warm-up 1 (worked) — A first Makefile, felt#

Build an output from an input, re-run (up to date), change the input, re-run (rebuilds). Mind the TAB.

wc -l < input.txt > count.txt
make: 'count.txt' is up to date.
wc -l < input.txt > count.txt
--- count.txt ---
4

✓ make skipped the build when fresh and re-ran the recipe after the input changed

Warm-up 2 (your turn) — A variable and a `.PHONY clean`#

Rewrite the rule to name the output via a variable, and add a .PHONY clean target that deletes it. Run make, then make clean.

wc -l < input.txt > count.txt
built:
count.txt
rm -f count.txt
after clean:
ls: cannot access 'count.txt': No such file or directory
count.txt removed

✓ the variable named the output, make built it, and .PHONY clean removed it

Applied 1 (your turn) — Automatic variables and a dry run#

Write a rule that uses $@ and $< instead of repeating filenames, then preview it with make -n before building.

sort names.txt > sorted.txt
sort names.txt > sorted.txt
--- sorted.txt ---
alpha
beta
gamma

✓ make -n previewed the resolved recipe (sort names.txt) without building, then make sorted correctly

Applied 2 (worked) — A pattern rule, incremental#

One %.upper: %.txt rule uppercases every .txt. Build all, then change one input and watch only that one rebuild.

tr a-z A-Z < a.txt > a.upper
tr a-z A-Z < b.txt > b.upper
tr a-z A-Z < c.txt > c.upper
--- first build done ---
tr a-z A-Z < b.txt > b.upper
--- b.upper ---
BETA2

✓ the pattern rule rebuilt only the changed file (b), leaving a and c untouched

Composite — putting it together (the course pipeline)#

The grand synthesis. Build the Makefile that, for every trajectory, extracts its minimum energy and aggregates the results into a table, composing globs (Notebook 5), extraction (Notebooks 6/8), automatic vars and .PHONY (Notebook 12-ish), incremental rebuilds, and a parallel-capable build. Then exercise it: make, change one file and re-make, and make clean.

grep 'E =' run1.xyz | awk '{print $NF}' | sort -n | head -1 > run1.energy
grep 'E =' run2.xyz | awk '{print $NF}' | sort -n | head -1 > run2.energy
grep 'E =' run3.xyz | awk '{print $NF}' | sort -n | head -1 > run3.energy
for f in run1.energy run2.energy run3.energy; do printf '%s\t%s\n' "${f%.energy}" "$(cat $f)"; done > results.txt
--- results.txt ---
run1	-12.3
run2	-22.4
run3	-7.7
rm -f *.energy results.txt
--- after clean ---
ls: cannot access '*.energy': No such file or directory
ls: cannot access 'results.txt': No such file or directory
all build products removed

✓ the pipeline built the 3-row table, rebuilt only the changed trajectory, and clean reset it

Optional stretch — Parallel timing, or the per-line-shell gotcha#

No grade. Two directions. (a) Time a full rebuild serially versus in parallel: make -j runs independent recipes at once (Notebook 17’s speedup, now for builds):

serial:
real	0m0.021s
user	0m0.017s
sys	0m0.014s
parallel:
real	0m0.020s
user	0m0.007s
sys	0m0.024s
(on this tiny pipeline the times are dominated by noise — the point is -j runs them at once)

(b) Or feel the “each recipe line is its own shell” gotcha: a cd on one line does not persist to the next, so chain with &&. Try writing a two-line recipe where the second line assumes the first’s cd (and watch it not work) then fix it with &&.

Outlook — the end of the course#

That is the whole course. You began perhaps having never opened a terminal; you can now move through a filesystem, find and pull what you need out of files, edit and write robust scripts, take work to a cluster and reason about its resources, and tie the whole pipeline together with a single make. The very tool in this last notebook is the one that builds the website these lessons live on: you have, quite literally, learned how the course is made.

From here the path is the work this was always a preparation for: the Molecular and Materials Modelling course, and the real research that lives on the command line. Keep the Compendium Scriptorum close: it is the map of every command you have met, and where to find each again. You are ready. Go build something.

New in the Compendium

make — run a Makefile — build each target from its prerequisites, rebuilding only what is out of date.

See the full Compendium Scriptorum for every command met so far, and where to find it again.