7 The stream toolkit

7 The stream toolkit#

What this notebook is about#

grep handed you the matching lines. These five tools reshape them: turning a pile of lines into a count, a single column, an ordered list, a tally. Each does exactly one thing, reads from stdin and writes to stdout, and is built to sit in a pipe. (A column of energies is just a column of numbers, as ever: no physics needed.)

This is the notebook where the Unix philosophy finally pays off in full. Until now the pipelines were short, because there was little to chain. With five reshaping filters in hand, real multi-stage pipelines arrive, and the chapter ends on the single most useful one in all of shell work.

A. `wc` — how much is there?#

The simplest question: how big? wc counts lines, words, and bytes.

wc — count the lines, words, and bytes of its input.

-l	count lines
-w	count words
-c	count bytes

more: man wc

Watch out: -l counts *newlines*, so a file whose last line has no trailing newline is counted one short

wc -l data/trajectories/lj38-optimization.xyz

2520 data/trajectories/lj38-optimization.xyz

Two and a half thousand lines — far too many to read, which is the whole reason we filter. Give wc a file and no flag and it prints all three counts at once:

wc data/logs/gr2hno3-nvt.log

  721  3336 42963 data/logs/gr2hno3-nvt.log

B. `cut` — pick a column#

Lines are often columns of data. cut keeps the columns you want. Those CP2K log lines are conveniently TOKEN| …, so the | is a clean delimiter: -d'|' sets it, -f1 keeps the first field: the section name on each line.

cut — pick out columns from each line — by delimited field, or by character position.

-d C	use C as the field delimiter (default is TAB)
-f N	keep field N (or a list/range like `1,3` or `2-4`); use with `-d`
-c N	keep character positions N instead of fields (e.g. `1-6`)

more: man cut

Watch out: the delimiter is a SINGLE character; cut cannot cope with runs of variable whitespace — that is exactly where awk takes over (Notebook 8)

grep '|' data/logs/gr2hno3-nvt.log | cut -d'|' -f1 | head -n 5

 DBCSR
 DBCSR
 DBCSR
 DBCSR
 DBCSR

cut can also slice by character position with -c, ignoring fields entirely:

cut -c1-9 data/logs/gr2hno3-nvt.log | head -n 5

 DBCSR| C
 DBCSR| M
 DBCSR| M
 DBCSR| M
 DBCSR| M

There is a wall here, though, and it matters because it points straight at the next notebook. cut’s delimiter is a single character, and it counts every space as a field separator. The energy value is padded away from its label by a run of spaces, so those spaces become a dozen empty fields and the value is marooned out at field 22. Reach for it at any smaller, predictable field and you land on the wrong column — here, the unit label:

grep -m1 'Total FORCE_EVAL' data/logs/gr2hno3-nvt.log | cut -d' ' -f9

(a.u.):

That is the label (a.u.):, not the energy — cut reaching its limit. The moment columns are separated by variable whitespace, you want awk, which is the first half of Notebook 8.

C. `sort` — put it in order#

sort orders lines. Pull the energies out (the Notebook 6 way), and order them:

sort — order the lines of its input.

-n	numeric order (treat the field as a number, not text)
-r	reverse the order
-k N	sort by field N (with `-t` to set the separator)
-t C	use C as the field separator for `-k`
-u	drop duplicate lines, keeping one of each

more: man sort

Watch out: the DEFAULT is lexical (dictionary) order, so 10 sorts before 2 and -9 before -10 — reach for -n whenever the column is numbers

grep 'Total FORCE_EVAL' data/logs/gr2hno3-nvt.log | grep -oE '\-[0-9]+\.[0-9]+' | sort -n

-145.725465772527684
-144.026841371125641
-143.350064790371448
-142.878629961526173
-142.438732194981611
-142.246533543175843

Note the -n. Leave it off and you meet the flagship trap of this notebook. By default sort compares lines as text, character by character, so it puts the energies in the wrong order, because lexically -142… comes before -145… (the 2 beats the 5), even though -145 is the smaller number:

grep 'Total FORCE_EVAL' data/logs/gr2hno3-nvt.log | grep -oE '\-[0-9]+\.[0-9]+' | sort

-142.246533543175843
-142.438732194981611
-142.878629961526173
-143.350064790371448
-144.026841371125641
-145.725465772527684

The two orders are reversed. The textbook version of the same bug: lexically, 10 sorts before 2 (because 1 beats 2). Whenever a column is numbers, reach for -n.

D. `uniq` — collapse duplicates#

uniq folds repeated lines into one, and with -c counts how many there were. Counting how often each section appears in the log is a natural fit, but watch what happens if we forget one thing:

uniq — collapse adjacent duplicate lines, and optionally count them.

-c	prefix each line with the number of times it occurred
-d	print only the lines that were duplicated
-u	print only the lines that were never repeated

more: man uniq

Watch out: uniq only compares *adjacent* lines, so duplicates scattered through the file are missed — almost always run sort | uniq

grep '|' data/logs/gr2hno3-nvt.log | cut -d'|' -f1 | tr -d ' ' | uniq -c

DBCSR
CP2K
GLOBAL
MEMORY
EWALD
MD
ROT
THERMOSTAT
ENERGY
MD_ENERGIES
ENERGY

Look closely: ENERGY is counted twice, as 1 and then 5. That is the number-one uniq confusion: it only looks at adjacent lines. A section that appears in two different places in the log shows up as two separate runs, each counted on its own. The fix is to sort first, so all the identical lines are brought together:

grep '|' data/logs/gr2hno3-nvt.log | cut -d'|' -f1 | tr -d ' ' | sort | uniq -c

CP2K
DBCSR
ENERGY
EWALD
GLOBAL
MD
MD_ENERGIES
MEMORY
ROT
THERMOSTAT

Now each section is counted once, correctly. sort | uniq is one of the most common two-command phrases in the shell: almost any time you reach for uniq, sort comes just before it.

E. `tr` — transform characters#

The last filter works on characters, not lines. tr translates one set of characters to another, deletes a set, or squeezes runs of a character to one.

tr — translate, delete, or squeeze characters in a stream (it reads standard input, so pipe into it).

SET1 SET2	translate each character of SET1 to the matching one of SET2 (e.g. `'a-z' 'A-Z'` upper-cases)
-d SET	delete every character in SET (e.g. `-d '\r'` strips carriage returns)
-s SET	squeeze runs of a SET character down to one (e.g. `-s ' '` collapses spaces)

more: man tr

Watch out: tr works on character SETS, not strings or patterns — string and regex replacement is sed (Notebook 8)

Upper-casing is the textbook example: translate the lowercase range to the uppercase one:

echo "total energy" | tr 'a-z' 'A-Z'

TOTAL ENERGY

-s squeezes runs down to one, which tidies the ragged whitespace cut choked on earlier (a workaround, not a replacement for awk):

grep -m1 'Total FORCE_EVAL' data/logs/gr2hno3-nvt.log | tr -s ' '

 ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -142.246533543175843

And -d deletes: the everyday use is stripping the stray carriage returns that Windows leaves on line ends (\r):

printf 'a line\r\n' | tr -d '\r' | cat -A

a line$

The ^M is gone, leaving a clean $ line end. Note the scope boundary, the twin of cut’s: tr maps character sets. It cannot replace a string or a pattern; that is sed, the other half of Notebook 8.

Exercises#

A pipeline-heavy set, because that is the point of these tools. All reads are read-only and deterministic; the one exercise that saves a result writes into a fresh scratch/.

Warm-up 1 (worked) — Count#

Count the lines of a trajectory and of a log with wc -l, and the words of the log with -w.

2520 data/trajectories/lj38-optimization.xyz
721 data/logs/gr2hno3-nvt.log

✓ the trajectory is 2520 lines and the log is 721

Warm-up 2 (your turn) — Cut a column#

The log’s lines are TOKEN| …. Use cut with -d and -f to keep just the section token on every | line, and count how many distinct sections there are (pipe through sort -u).

CP2K
DBCSR
ENERGY
EWALD
GLOBAL
MD
MD_ENERGIES
MEMORY
ROT
THERMOSTAT

✓ cut pulled the token column; there are 10 distinct sections

Applied 1 (your turn) — Sort numerically#

Extract the energy column (grep -oE, from Notebook 6) and order it with sort -n. The first line is then the lowest (most negative) energy. Try it once without -n to watch the lexical mangling.

-145.725465772527684

✓ sort -n found the lowest energy, -145.72…

Applied 2 (your turn) — Dedup and count#

Count how many lines belong to each log section: sort the token column, then uniq -c. Confirm that the DBCSR section is the busiest. (Remember: sort before uniq.)

CP2K
DBCSR
ENERGY
EWALD
GLOBAL
MD
MD_ENERGIES
MEMORY
ROT
THERMOSTAT

✓ sort | uniq -c counted DBCSR's 23 lines

Applied 3 (worked) — `tr` cleanup#

Three character transforms: squeeze the ragged spaces of an energy line, strip a carriage return, and lower-case a stream.

 ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -142.246533543175843
value$
warning

✓ spaces squeezed, carriage return stripped, and the line lower-cased

Composite — putting it together (the frequency idiom)#

The payoff. Build the single most useful pipeline in shell text-work, the ranked-frequency idiom, to rank the log’s sections from busiest to quietest:

grep '|' … | cut -d'|' -f1 | tr -d ' ' | sort | uniq -c | sort -rn

It chains grep (Notebook 6) with four of this notebook’s filters and the pipe (Notebook 4): select the | lines, cut the token, tidy it, sort so duplicates are adjacent, count them with uniq -c, and finally sort -rn to rank by count, biggest first. Save the ranking to scratch/.

DBCSR
GLOBAL
MD
CP2K
THERMOSTAT
MEMORY
ROT
ENERGY
EWALD
MD_ENERGIES

✓ the ranking puts DBCSR (23 lines) at the top

Optional stretch (your turn) — A deeper pipeline#

Go one stage further. Take the energy column, sort -n, and keep the three lowest with head -n 3: the three most-bound configurations. Then, for the boundary lesson: try to pull the second numeric column out of an energy line with cut and watch the ragged whitespace defeat it. That failure is precisely the case for awk, next notebook.

-145.725465772527684
-144.026841371125641
-143.350064790371448

✓ the three lowest energies came out in numeric order, lowest (-145.72…) first

Outlook#

You can now reshape streams into counts, columns, and ordered tables, and you have hit two walls on purpose. cut cannot handle ragged whitespace, and tr cannot replace strings. Next (Notebook 8): awk and sed (field-aware processing and stream editing), which clear both walls at once and carry the regular-expression thread from Notebook 6 right through to the end of Part II.

New in the Compendium

wc — count the lines, words, and bytes of its input.
cut — pick out columns from each line — by delimited field, or by character position.
sort — order the lines of its input.
uniq — collapse adjacent duplicate lines, and optionally count them.
tr — translate, delete, or squeeze characters in a stream (it reads standard input, so pipe into it).

See the full Compendium Scriptorum for every command met so far, and where to find it again.