6 grep and regular expressions#
What this notebook is about#
Notebook 5 found files by their names. This notebook looks inside them, for
text, and it does so by pattern, not by exact string. The tool is grep;
the pattern language is regular expressions.
Those two are not equally important. grep is one command among many. Regular
expressions are the most reusable text skill in this whole course: the same
patterns drive sed in Notebook 8 and the Vim search-and-replace from Notebook 9.
Learn them once here and they pay off everywhere. So we teach both: the tool, and
the language behind it.
The files are real CP2K run logs now living in data/logs/. Matching total
energy or a warning in one of them needs no physics: it is just pattern on
text.
A. grep with literal patterns#
At its simplest, grep PATTERN FILE prints every line of the file that contains
the pattern. Start literal: find the total-energy lines in a run log.
| -i | match case-insensitively |
| -v | invert — print the lines that do NOT match |
| -n | prefix each match with its line number |
| -c | print only a count of matching lines |
| -r | recurse through a directory tree |
| -l | print only the names of files that contain a match |
| -o | print only the matched part of each line, not the whole line |
| -E / -F | -E extended regex (the sane quantifiers/grouping); -F fixed string (the pattern is taken literally) |
man grep — it has more flags than fit here; this is also where you find -A/-B/-C (context)grep '^Total' file) so the shell does not expand $, *, etc. before grep sees it (the same shell-expands-first lesson as globs); and grep's DEFAULT is the quirky Basic regex, where + ? ( ) | { } are literal — reach for -E to get the Extended regex this notebook teachesgrep 'Total FORCE_EVAL' data/logs/gr2hno3-nvt.log
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -142.246533543175843
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -142.438732194981611
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -142.878629961526173
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -143.350064790371448
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -144.026841371125641
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -145.725465772527684
Six lines: the energy after each step of the run. Note the single quotes
around the pattern: get into that habit now. They stop the shell from touching the
pattern (expanding a $, a *, and so on) before grep ever sees it, exactly
the “the shell goes first” lesson from globbing, and a habit that becomes essential
the moment your patterns contain regex characters.
The workhorse flags turn a match into an answer. Just count the matches with
-c:
grep -c 'Total FORCE_EVAL' data/logs/gr2hno3-nvt.log
6
Number the matches with -n, case-fold with -i (the log shouts WARNING in
capitals; -i catches it however it is spelled):
grep -in 'warning' data/logs/gr2hno3-nvt.log
168: *** WARNING in motion/simpar_methods.F:223 :: A temperature tolerance ***
704: *** WARNING in dbcsr_mm.F:295 :: Using a non-square number of MPI ranks ***
714: The number of warnings for this run is : 2
And -l answers “which files contain this?” without printing the matches,
indispensable across a directory of logs:
grep -l 'Total FORCE_EVAL' data/logs/*.log
data/logs/gr2hno3-nvt.log
data/logs/gr2hno3-restart.log
B. Regular expressions#
A literal pattern is the floor. The ceiling is a regular expression: a small
language for describing shapes of text, such as “a line starting with ENERGY”, “a
signed decimal number”, “either warning or error”. We teach the Extended
flavour (ERE), which you reach with grep -E. Here is the workhorse set.
grep -E (and later sed -E and Vim). Curated essentials, not a dictionary.
| ^ $ | anchor to the start / end of the line |
| . | any single character |
| [abc] [^abc] | one character in / not in the set; ranges like [0-9] |
| [[:digit:]] [[:space:]] | named POSIX classes (a digit; whitespace) |
| * + ? | the preceding item: zero-or-more / one-or-more / zero-or-one |
| {n,m} | between n and m repetitions |
| a|b | alternation: match a or b |
| ( … ) | group, e.g. to apply a quantifier or alternation to several characters |
| \. | backslash escapes a metacharacter to mean it literally (a real dot) |
Read one slowly, piece by piece: the pattern that matches a signed decimal like the energies above:
-? [0-9]+ \. [0-9]+ matches e.g. -142.246533
│ │ │ │
│ │ │ └─ [0-9]+ one or more digits (the decimals)
│ │ └─────── \. a literal dot (escaped — not "any char")
│ └───────────── [0-9]+ one or more digits (before the dot)
└─────────────────── -? an optional minus ( ? = zero or one )
Watch it work. Anchored to the line start, “ENERGY after any leading spaces”
finds the energy lines (^, the POSIX space class, and a quantifier together):
grep -E '^[[:space:]]*ENERGY' data/logs/gr2hno3-nvt.log
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -142.246533543175843
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -142.438732194981611
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -142.878629961526173
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -143.350064790371448
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -144.026841371125641
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -145.725465772527684
Alternation pulls warnings or errors in one pass, the everyday log-triage move:
grep -iE 'warning|error' data/logs/gr2hno3-nvt.log
*** WARNING in motion/simpar_methods.F:223 :: A temperature tolerance ***
*** WARNING in dbcsr_mm.F:295 :: Using a non-square number of MPI ranks ***
The number of warnings for this run is : 2
ERE vs BRE, and what we are skipping
Reach for grep -E. Without it, grep uses the older Basic regex (BRE), in
which + ? { } ( ) | are literal characters unless you backslash them, a
reliable source of “why doesn’t my pattern work”. -E gives the Extended syntax
above, and the same -E works for sed later. Two things we deliberately leave
out as beyond a prerequisite course: Perl-style grep -P (PCRE), and
backreferences / lookaround. They exist; you do not need them yet.
C. Pattern and flag together, on a real log#
The power is in combining a regex with the flags. The signed-decimal pattern, fed
the energy lines, and then -o to print only the matched part: the numbers
themselves, stripped of the surrounding text:
grep 'Total FORCE_EVAL' data/logs/gr2hno3-nvt.log | grep -oE '\-[0-9]+\.[0-9]+'
-142.246533543175843
-142.438732194981611
-142.878629961526173
-143.350064790371448
-144.026841371125641
-145.725465772527684
That is a first taste of extraction: turning lines into values. Mind the
scope boundary, though: grep selects (and with -o, clips) text; it does not
do columns or arithmetic. When you want “the third field of every line”, that is
awk, in Notebook 8. Here, grep finds; it does not transform.
D. grep -r, and how it differs from find#
Everything so far searched one file. -r searches a whole tree, descending
into every file under a directory. “Which logs mention a warning, and how often?”
grep -rc 'WARNING' data/logs
data/logs/gr2hno3-restart.log:2
data/logs/gr2hno3-nvt.log:2
This is the natural place to draw a line you have been circling since Notebook 5.
Both find and grep -r walk a directory tree, but they look for different
things:
findlocates files, by their name and metadata (find data -name "*.log").grep -rlocates text, by what is inside the files (grep -r 'WARNING' data).
And, being good Unix citizens, they compose: find the files you care about, then
grep their contents, a pairing you will reach for constantly.
Exercises#
Regular expressions reward practice, so this is a full set: literal grep to warm
up, then ERE on the real logs, the course’s dedicated look-it-up exercise, and a
capstone that mines a log end to end. grep only reads; the one exercise that
saves its results writes into a fresh scratch/.
Warm-up 1 (worked) — Match, count, locate#
Find the total-energy lines in the run log, count them with -c, and use -l to
see which of the two logs contain them.
6
data/logs/gr2hno3-nvt.log
data/logs/gr2hno3-restart.log
✓ six energy lines counted, and both logs contain them
Warm-up 2 (your turn) — Invert#
Use -v to print the lines that do not match. Count the non-blank lines of
the log by inverting a pattern that matches blank lines ('^[[:space:]]*$': start,
any spaces, end).
571
✓ the inverted count is exactly the 571 non-blank lines
Applied 1 (your turn) — Anchors and classes#
Write ERE patterns (grep -E) for two things in
data/trajectories/lj38-optimization.xyz: the comment lines, which start with
whitespace then i = (use ^ and a POSIX class), and count them; there is one
per frame.
i = 1, E = -0.0231086608
i = 4, E = -0.0279588737
i = 7, E = -0.0287650150
63
✓ the anchored pattern matched all 63 frame-comment lines
Applied 2 (your turn) — Quantifiers and alternation#
Two patterns on the run log: count the lines that are a warning or an error
(-iE 'warning|error'), and confirm the signed-decimal pattern
-?[0-9]+\.[0-9]+ appears on the energy lines.
3
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -142.246533543175843
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -142.438732194981611
ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -142.878629961526173
✓ three warning/error lines, and the signed-decimal pattern matched all six energy lines
Applied 3 (worked) — Extract the matched part#
Pull just the numbers out of the energy lines with -o: a first taste of
extraction (the full version, with fields, is awk in Notebook 8).
-142.246533543175843
-142.438732194981611
-142.878629961526173
-143.350064790371448
-144.026841371125641
-145.725465772527684
✓ six numbers extracted, one per energy line
Discovery (your turn) — Find the flag yourself#
This is the course’s standing skill: needing a flag you were never handed, and
finding it. Your task: print three lines of context around each WARNING in
the log: the warning and its neighbours. That flag is not on the card. Consult
grep --help or man grep, find the context option (-C, or -A/-B), and use
it.
MD| Dump 1000 gr2hno3_nvt-1.restart
*** WARNING in motion/simpar_methods.F:223 :: A temperature tolerance ***
*** (TEMP_TOL) is used during the MD. Due to the velocity rescaling ***
*** algorithm jumps may appear in the conserved quantity. ***
--
... [further MD steps elided for the course excerpt] ...
-------------------------------------------------------------------------------
*** WARNING in dbcsr_mm.F:295 :: Using a non-square number of MPI ranks ***
✓ the context flag printed more than just the matching lines
Composite — putting it together (capstone)#
Mine the run log end to end and save a small report, using regex, flags, a pipe
(Notebook 4), and redirection (Notebook 4). In scratch/: write the count of
SCF steps, the list of energy values, and the warnings with context into
three files.
6
-142.246533543175843
-142.438732194981611
-142.878629961526173
-143.350064790371448
-144.026841371125641
-145.725465772527684
✓ the report has 6 SCF steps, 6 energy values, and a non-empty warnings file
Optional stretch (your turn) — Recurse, and contrast with find#
Search across the tree with grep -r, then reach the same logs via find
(Notebook 5) and grep them — two routes to one answer. Here: count the energy lines
in every log under data/, recursively, then do it by piping find into grep.
(Aside, for Part IV: a big recursive grep is one of the jobs xargs -P can run in
parallel.)
data/logs/gr2hno3-restart.log:3
data/logs/gr2hno3-nvt.log:6
data/logs/gr2hno3-restart.log:3
data/logs/gr2hno3-nvt.log:6
✓ the two routes — grep -r and find | grep — reach the same matches
Outlook#
You can now find and pattern-match text, and you have the regular-expression
foundation that the rest of the course leans on. Next (Notebook 7): the stream
toolkit (cut, sort, uniq, wc, tr), the small tools that reshape the
lines grep selects, turning a pile of matches into counts, columns, and ordered
tables.
grep— print the lines of a file (or of its input) that match a pattern — literal text, or a regular expression.
See the full Compendium Scriptorum for every command met so far, and where to find it again.