10 Quoting and expansion

10 Quoting and expansion#

What this notebook is about#

You have met this idea three times already and never named it. In Notebook 5 the shell expanded *.xyz before ls ran. In Notebooks 6 and 8 you single-quoted your grep and awk programs so the shell would leave the $ alone. Both are the same fact: before any command runs, the shell rewrites your command line.

This notebook names that fact, lays out the full rewriting, and, most usefully, shows how quoting lets you control it. Getting this wrong is the single most common subtle bug in the shell, so it earns a whole notebook here at the start of Part III, right before you begin writing scripts of your own. (As ever: this is all filenames and values, no physics.)

A. A bug, to start#

Make a file whose name contains a space, and put its name in a variable:

f="scratch/my run.xyz"

Now look closely. We pass $f, unquoted, to a command, and print each argument the command actually receives:

printf 'argument: [%s]\n' $f

argument: [scratch/my]
argument: [run.xyz]

The command received two arguments, not one. The space in the filename tore $f into scratch/my and run.xyz. Try to use it and it breaks: cat goes looking for two files that do not exist:

cat $f 2>/dev/null || echo '(failed — the unquoted variable split into two names)'

(failed — the unquoted variable split into two names)

Hold that bug in mind. By the end of the notebook it will be a one-character fix you never forget.

B. The model: the shell rewrites your line#

Here is what really happens. When you press Enter, the shell does not hand your line to the command as you typed it. It first rewrites the line through a fixed sequence of expansions, each one substituting something in, and only then runs the command on the result. You have already met most of them:

The expansions, in order — the shell applies these to your line before the command runs; the command sees only the result.

{a,b} {1..9}	brace expansion: generate strings (Notebook 5)
~	tilde: your home directory (Notebook 2)
$var ${var}	parameter expansion: the value of a variable
$(cmd)	command substitution: the output of a command
$((expr))	arithmetic expansion: integer arithmetic
word splitting	split the unquoted results on whitespace: the culprit in §A
* ? [ ]	pathname (glob) expansion: match filenames (Notebook 5)

The whole notebook hangs off the bold row. Word splitting is what chopped $f in two: it splits unquoted expansion results on spaces. Which means the fix is to stop the result from being unquoted. That is what quoting is for.

C. Quoting — the control#

Three tools, three behaviours:

Quoting — shell syntax (not a command) that controls which expansions happen.

'single'	literal: no expansion at all, the text is taken exactly as written
"double"	allow `$`-expansions, but suppress word splitting and globbing
\\ (backslash)	escape: protect the single next character

Single quotes turn everything off: the shell does not touch what is inside:

echo '$HOME and *.xyz are left exactly as typed'

$HOME and *.xyz are left exactly as typed

That is precisely why you single-quote grep and awk programs (Notebooks 6 and 8): a $1 inside single quotes reaches awk untouched. Double quotes are the middle ground: variables do expand, but the result is kept as one piece:

echo "your home is $HOME"

your home is /home/runner

And a backslash protects just the next character: here, a literal dollar sign:

echo "a price tag: \$5"

a price tag: $5

D. The one habit, and the traps#

Everything above points at a single rule, and it is the highest-value habit in all of shell scripting:

⚠ Always double-quote your variable expansions

Write "$f", not $f. An unquoted variable that happens to contain a space becomes two arguments (the §A bug); one that contains a * gets glob-expanded against the current directory, matching who-knows-what. Quietly, with no error, your command operates on the wrong thing, and when that command is rm, the result is the kind of story people tell years later. The fix costs two characters. Quote every "$var" and every "$@", every time, and a whole category of bugs simply never happens.

Watch the fix retire the §A bug: the same command, the only difference being the quotes:

printf 'argument: [%s]\n' "$f"

argument: [scratch/my run.xyz]

cat "$f"

frame data

One argument, the right file. A few more traps worth knowing while we are here:

Empty or unset variables vanish when unquoted, which can turn command $x into just command. A default value guards against it: ${x:-fallback} (more in §E).
"$@" is the right way to forward “all the arguments, each kept whole”; unquoted $@ and $* both word-split and will mangle any argument with a space. (You will use "$@" constantly once you write scripts in Notebook 12.)
Backticks `cmd` are the old form of command substitution. Use $(cmd) instead: it nests cleanly and reads better.

E. The workhorse `${ }` forms#

Parameter expansion does more than fetch a value. A curated handful of ${…} forms covers almost everything you will reach for, most of it about reshaping filenames, which is exactly what scripts spend their time doing.

Parameter expansion — the workhorse ${…} forms. Case conversion, substrings, indirection, and arrays are out of scope.

${var:-default}	use default if var is unset or empty
${#var}	the length of var
${var%.xyz}	remove a matching suffix (strip an extension)
${var#prefix}	remove a matching prefix
${var/old/new}	substitute the first old with new
$(cmd)	command substitution: capture a command's output
$((expr))	arithmetic: integer only

Suffix-stripping is the daily one: take an input name, drop its extension, build the matching output name. This is most of what a data script does between files:

name="lj38-relaxed.xyz"
echo "${name%.xyz}.dat"

lj38-relaxed.dat

Command substitution drops a command’s output into your line, and arithmetic does integer math:

count=$(ls data | wc -l)
echo "data/ holds $count entries"

data/ holds 6 entries

echo "17 divided by 5 is $(( 17 / 5 ))"

17 divided by 5 is 3

Note that 3: $(( )) is integer only, and it truncates (it does not round). The moment you need a real number (a mean, a ratio with decimals) the shell cannot help, and you reach back for awk (Notebook 8) or bc:

grep 'Total FORCE_EVAL' data/logs/gr2hno3-nvt.log | grep -oE '\-[0-9]+\.[0-9]+' | awk '{ s += $1; n++ } END { printf "mean = %.4f\n", s/n }'

mean = -143.4444

That is the honest division of labour: the shell for integers and string-shaping, awk for the floating-point.

Exercises#

The point of this set is to feel the bugs and fix them. Anything that creates a file works in a fresh scratch/; the rest is read-only.

Warm-up 1 (worked) — The three quote types#

echo a variable with no quotes, single quotes, and double quotes, and watch expansion switch off and on.

hello
hello from /home/runner
$greeting stays literal

✓ double quotes expanded the variable; single quotes kept it literal

Warm-up 2 (your turn) — Command substitution#

Capture the output of a command into a variable with $(…), then use it in a sentence. Count the entries in data/ and report the number.

data/ has 6 entries

✓ the command's output was captured into the variable

Applied 1 (your turn) — The word-splitting bug, then the fix#

The centrepiece. In scratch/, make a file whose name has a space. Operate on it through an unquoted variable and watch it fail; then quote the variable and watch it work.

unquoted: failed (split into two names)
content

✓ the unquoted form failed and the quoted form succeeded

Applied 2 (your turn) — The workhorse `${ }` forms#

Given a filename, build derived strings: strip the .xyz to make a .dat output name (${f%.xyz}), get the name’s length (${#f}), and supply a default for an unset variable (${u:-none}).

lj38-relaxed.dat
length is 16
label is none

✓ the output name, length, and default were all derived correctly

Applied 3 (worked) — Arithmetic and its limit#

Integer arithmetic with $(( )) (note the truncation), then a mean (which needs floats) with awk.

integer: 66
float mean: -143.4444

✓ integer division truncated to 66, and awk gave the float mean -143.4444

Composite — putting it together (build commands safely)#

Build output filenames from a set of trajectory files (one with a space in its name) using trimming and properly quoted expansions. (The for … in …; do … done loop here just means “for each file”; it gets its full introduction in Notebook 13. The lesson now is that quoting makes the loop space-safe.)

'run 3.dat'  'run 3.xyz'   run_1.dat   run_1.xyz   run_2.dat   run_2.xyz

✓ a .dat was built for every .xyz, including the one whose name has a space

Optional stretch (your turn) — An expansion-order puzzle#

A glob stored in a quoted variable does not expand, because globbing happens to the line, not to a quoted value. Confirm it: put *.xyz in a variable, then echo it quoted (stays literal) versus letting it expand. Predict each before running.

*.xyz
x.xyz y.xyz

✓ quoted, the pattern stayed literal; unquoted, it expanded to the two files

Outlook#

You now control how the shell reads what you type: the foundation that keeps a script from breaking the first time a filename has a space in it. That foundation matters because Part III is about turning your one-liners into scripts you keep and re-run. The next missing piece is mechanical: what makes a file runnable in the first place. Next (Notebook 11): permissions and execution.