8 awk and sed#
What this notebook is about#
Notebook 7 ended at two walls. tr could only swap characters, not replace
strings. cut could only split on a single character, and could not do
arithmetic. These two tools clear both walls and finish the extraction toolkit:
sededits a stream (find-and-replace and more), picking up the regular expressions from Notebook 6.awksplits each line into fields and can compute on them, which is the climax of Part II: doing actual arithmetic on the numbers in real data.
A word of honesty first. sed and awk are each whole programming languages;
people write hundred-line programs in them. We are not doing that. We take the
daily workhorse slice of each, name the deep parts as out of scope, and stop
there. That is the right amount for a shell primer, and it is genuinely most of
what you will ever type. (And, as always: averaging a column of energies is just
averaging a column of numbers: no physics required.)
A. sed — stream editing#
sed reads its input line by line, applies an editing command to each, and prints
the result. The mental model is simple: it is grep that can also change what it
finds. Its headline command is substitution, s/old/new/.
| -E | use Extended regex (the same ERE as grep -E); sed otherwise defaults to Basic regex |
| -n | suppress the automatic printing of every line; pair with the p command to print only what you choose |
| -i / -i.bak | edit the file IN PLACE; -i.bak keeps a backup of the original first (see the admonition) |
man sedsed 's/old/new/') so the shell does not touch $, &, etc.; and -i rewrites the file with NO undo — keep a .bakHere it is on a scratch config: replace every 300 with 500. Note sed prints
to the screen and leaves the file untouched (more on that below):
sed 's/300/500/g' scratch/run.in
TEMPERATURE 500
CUTOFF 500
MAX_STEPS 1000
# a trailing comment
The g made it global (every match on each line, not just the first). The regular
expression is the same language as Notebook 6 — reach for sed -E to get the
Extended syntax, since sed, like grep, defaults to the quirkier Basic regex.
With -E, & in the replacement stands for the whole matched text, handy for
wrapping or annotating a match:
sed -E 's/[0-9]+/<&>/g' scratch/run.in
TEMPERATURE <300>
CUTOFF <300>
MAX_STEPS <1000>
# a trailing comment
sed can also just select. With -n (don’t auto-print) and the p command, it
prints only matching lines, essentially grep:
sed -n '/CUTOFF/p' scratch/run.in
CUTOFF 300
And d deletes. Addressed by a regex, /^#/d strips comment lines; addressed by
number, 2,4d deletes a line range:
sed '/^#/d' scratch/run.in
TEMPERATURE 300
CUTOFF 300
MAX_STEPS 1000
The full sed command vocabulary is below. Beyond it lie the hold space,
branches, and multi-command scripts: real, formidable, and firmly out of scope
for this course. When you need them, you will know, and you will reach for a
reference.
-E for ERE). The deep parts (hold space, branching) are out of scope.
| s/old/new/ | substitute the first match on each line |
| s/old/new/g | substitute every match on the line |
| s/old/new/2 | substitute only the 2nd match; add I for case-insensitive |
| & | in the replacement, the whole matched text |
| -n /pat/p | print only lines matching pat (like grep) |
| /pat/d 2,5d $d | delete: lines matching pat / lines 2–5 / the last line |
⚠ sed -i edits in place — no undo
Everything above printed to the screen and left the file alone. Add -i and
sed rewrites the file in place, and, like rm and >, with no undo. A
substitution with a slightly-too-greedy pattern, run with -i, has quietly mangled
many a config. Two habits, the same family as the earlier warnings: dry-run
without -i first and read the output, and when you do commit, use -i.bak,
which saves the original as file.bak before editing. Never sed -i something you
have not first seen the result of.
sed -i.bak 's/300/500/g' scratch/run.in
ls scratch
run.in run.in.bak
The edit happened, and run.in.bak holds the original: your safety net.
B. awk — fields and computation#
awk also reads line by line, but it does something sed does not: it splits
each line into fields, $1, $2, … up to $NF (the last), and lets you act on
them. Crucially, it splits on any run of whitespace by default, which is exactly
the wall cut hit. Where cut -d' ' miscounts ragged spacing, awk just works:
| -F C | set the field separator to C (the default is any run of whitespace — which is how it beats cut) |
| -v var=val | pass a shell value into the program as an awk variable |
man awk$1, $2 look exactly like shell positional parameters, and the shell will eat them otherwise (the shell-expands-first thread again)grep -m1 'Total FORCE_EVAL' data/logs/gr2hno3-nvt.log | awk '{print $NF}'
-142.246533543175843
$NF pulled the energy (the last field) no matter how many spaces padded the
line. That single fact retires cut for anything real. The structure of an awk
program is pattern { action }: for every line matching pattern, run action.
Either part is optional. A bare condition filters; a bare action runs on every
line. NR is the current line number, so NR==1 selects the first line; a
field condition works the same way, so $1=="Kr" selects the krypton row:
printf 'Ar 1.5 2.5\nAr 3.5 4.5\nKr 5.5 6.5\n' | awk '$1=="Kr" {print $2, $3}'
5.5 6.5
Now the payoff, and the reason awk is the climax of Part II: it can compute.
A BEGIN block runs before the first line, an END block after the last, and in
between you can accumulate. Summing and averaging a column is the canonical move:
here, the six energies from the log:
grep 'Total FORCE_EVAL' data/logs/gr2hno3-nvt.log | grep -oE '\-[0-9]+\.[0-9]+' | awk '{ sum += $1; n++ } END { printf "mean = %.4f over %d values\n", sum/n, n }'
mean = -143.4444 over 6 values
That is real arithmetic on real numbers, in one line: the thing no tool before
this notebook could do. The workhorse awk vocabulary is below; arrays,
user-defined functions, and getline are the out-of-scope deep end.
| $0 $1 … $NF | the whole line; field 1 … the last field (NF = field count) |
| NR | the current record (line) number |
| pattern { action } | run action on each line matching pattern (either part optional) |
| /regex/ $3>0 NR==1 | patterns: a regex, or a condition on a field or NR |
| print printf | output a line, or formatted output |
| BEGIN{…} END{…} | run once before the first / after the last line: where totals live |
Single-quote your programs
Always wrap a sed or awk program in single quotes: awk '{print $1}'. Those
$1, $2 look exactly like shell positional parameters, and in double quotes the
shell would replace them before awk ever ran, usually with nothing. This is the
same “the shell goes first” rule from globbing and grep, and it is the single most
common awk/sed mistake.
Exercises#
A full set, climbing to the canonical trajectory task. Every sed edit works on a
fresh scratch/ copy (data/ is never touched), and the notebook gives the same
result every run.
Warm-up 1 (worked) — sed substitute and select#
On a scratch copy: globally substitute, then use -n '/pat/p' to print only
matching lines.
ALPHA 1
beta 2
ALPHA 3
gamma 4
beta 2
✓ both alphas were upper-cased and only the beta line was selected
Warm-up 2 (your turn) — sed delete by address#
On a scratch copy, delete the comment lines (those starting with #) with
/^#/d, and separately delete the first two lines with 1,2d.
keep one
keep two
keep three
keep two
# mid comment
keep three
✓ the two comment lines are gone, and 1,2d left exactly the last three lines
Applied 1 (your turn) — awk fields clear the wall#
The energy line is padded with ragged spaces, so cut -d' ' can’t reliably grab a
column. Show that, then pull the energy (the last field) with awk.
ENERGY| Total FORCE_EVAL
-142.246533543175843
✓ awk pulled the energy field where cut could not
Applied 2 (your turn) — awk patterns#
From the trajectory’s comment lines, print only the frames whose energy (the last
field) is below -0.06. (Pattern: a condition on $NF.)
91, -0.0601988256
94, -0.0604130301
97, -0.0609348186
100, -0.0610322312
103, -0.0613274365
106, -0.0613719574
109, -0.0615142731
112, -0.0615676636
115, -0.0616540453
118, -0.0617197984
121, -0.0617592599
124, -0.0618209437
127, -0.0618432759
130, -0.0618953104
133, -0.0619084439
136, -0.0619586363
139, -0.0619712325
142, -0.0620106315
145, -0.0620252547
148, -0.0620507388
151, -0.0620708540
154, -0.0620797550
157, -0.0620917993
160, -0.0620934610
163, -0.0620980889
166, -0.0620987337
169, -0.0621008468
172, -0.0621013073
175, -0.0621023237
178, -0.0621027034
181, -0.0621033425
184, -0.0621037945
186, -0.0621038220
✓ the condition split the frames into below and not-below -0.06
Applied 3 (worked) — awk arithmetic#
Sum and average a column with an END block: the six log energies.
sum=-860.6663 mean=-143.4444
✓ awk computed the mean energy, -143.4444
Composite — putting it together (the canonical task)#
The move you came to Part II for. From a real trajectory, extract the energy of
every frame (the comment line’s last field), then compute the mean energy, the
minimum, and the frame at which the minimum occurs, all in one awk pass
over the lines grep selects. Composes grep (NB6), awk, and the pipe (NB4).
mean = -0.054455
min = -0.062104 (frame 186)
✓ mean -0.054455, minimum -0.062104, occurring at frame 186
Optional stretch (your turn) — Safe in-place edit#
Do a real in-place edit, the safe way. On a scratch copy, use sed -i.bak to make
a substitution, then confirm the .bak backup holds the original and the file
holds the change. The point is the habit: never -i without a backup you have
checked.
cutoff = 500
cutoff = 300
✓ the edit took effect in the file, and the .bak preserved the original
Outlook#
You can now extract and compute from any text file the course holds, by hand, one one-liner at a time. That last phrase is the whole point of what comes next: you have been retyping these pipelines. Part III is about making them reusable (writing and editing them as scripts you keep), and it starts with the one tool you still lack: a way to write text in the terminal. Next (Notebook 9): editing with Vim.
sed— edit a stream line by line — substitute, delete, or selectively print — like grep that can also change what it matches.awk— split each line into fields and act on them — print columns, filter by condition, and compute (sum, mean, min/max). The field-aware, can-do-arithmetic tool cut could not be.
See the full Compendium Scriptorum for every command met so far, and where to find it again.