4 Redirection and pipes#
What this notebook is about#
Part I got you to a file and let you look inside it. Part II is about processing a file without reading it by eye, and this notebook is the plumbing that makes that possible.
The organising idea is worth saying plainly, because the rest of
the course hangs off it. The Unix way is many small tools that each do one thing
well, snapped together so the output of one becomes the input of the next. That
is why the coming notebooks teach grep, cut, sort, and awk as separate
little programs rather than one giant Swiss-army command: each is a pipe fitting,
and this notebook is the pipe.
We start mining the course’s real data/ files now. As always, though, you only
ever need to see them as text.
A. The three streams#
Every command, when it runs, is wired to three streams. Picture the command as a box:
stdin (standard input, stream
0): where it reads its input from;stdout (standard output, stream
1): where it writes its results;stderr (standard error, stream
2): where it writes its complaints.
┌─────────────────┐
stdin (0) ───────────▶ │ │ ──────▶ stdout (1) the results
a file, a pipe, or │ command │
what you type │ │ ──────▶ stderr (2) the errors
└─────────────────┘
Two results land in two different places on purpose: keeping errors (stderr) separate from results (stdout) is what lets you, say, save a command’s output to a file while its error messages still reach your eyes. The whole notebook is about steering these three streams.
B. Redirection — steer a stream to (or from) a file#
By default stdout and stderr both go to your screen and stdin comes from the
keyboard. Redirection points a stream somewhere else with a handful of
operators. The most common: > sends stdout into a file.
echo "the first line" > scratch/notes.txt
That printed nothing to the screen because its output went into the file instead. Read it back:
cat scratch/notes.txt
the first line
> overwrites. To add to a file instead, use >>:
echo "a second line" >> scratch/notes.txt
cat scratch/notes.txt
the first line
a second line
The mirror image is <, which feeds a file into a command’s stdin. Most
commands also take a filename directly, so you meet < less often, though it is
the same idea running backwards:
cat < scratch/notes.txt
the first line
a second line
⚠ > clobbers — it destroys before it writes
> does not ask. The instant you run cmd > file, the file’s old contents are
gone, replaced by the new output, even if cmd then fails and writes nothing.
This is the data-loss footgun of the notebook, and > important.txt in the wrong
moment has erased many a file. Two habits: when you mean to add, reach for >>,
not >; and before you > onto an existing file, glance at it (ls, cat) so you
know what you are about to overwrite.
Steering errors#
Results and complaints travel separately, so you can redirect them separately.
2> sends stderr to a file. Here is a command that does two things at once:
list a directory that exists and one that does not. The listing is stdout; the
“No such file” message is stderr.
(One bit of housekeeping: these commands deliberately fail, so we add || true at
the end to tell this notebook’s checker the error was on purpose. In your own
terminal you would just run the command.)
ls data nope-not-here || true
ls: cannot access 'nope-not-here': No such file or directory
data:
README.md inputs logs results scaling trajectories
Both streams landed on the screen, mixed together. Now send the errors into a file
with 2>, and only the results remain on screen:
ls data nope-not-here 2> scratch/errors.txt || true
data:
README.md inputs logs results scaling trajectories
The complaint went into the file:
cat scratch/errors.txt
ls: cannot access 'nope-not-here': No such file or directory
Often you do not want the errors at all. Redirect them to /dev/null, the
system’s black hole; anything sent there vanishes:
ls data nope-not-here 2> /dev/null || true
data:
README.md inputs logs results scaling trajectories
Clean: just the listing. Two more operators round out the set. 2>&1 means “send
stderr to wherever stdout is currently going” (handy for capturing both into one
place), and &> is the shorthand for “both streams to this file”. They are in the
reference table below.
Note
A subtlety to file away, not to dwell on: in cmd > file 2>&1 the order matters:
2>&1 copies stderr to wherever stdout points at that moment, so stdout must be
redirected first. Reverse them and stderr still goes to the screen. You will rarely
need this, but when a captured log is mysteriously missing its errors, this is why.
C. Pipes — the point of it all#
Redirection moves a stream to or from a file. The pipe, |, does something
better: it wires one command’s stdout straight into the next command’s
stdin, with no file in between. That is how small tools become large ones.
┌────────────┐ stdout ┌────────────┐ stdout ┌────────────┐
│ ls -l │ ────│────▶ │ head │ ────│────▶ │ ... │ ──▶ screen
└────────────┘ │ └────────────┘ │ └────────────┘
the pipe │ carries only stdout — errors still go to the screen
You have already seen a pipe, back in Notebook 1, without it being explained. The
line that fetched the top of the ls manual,
man ls | col -b | head -n 8
LS(1) User Commands LS(1)
NAME
ls - list directory contents
SYNOPSIS
ls [OPTION]... [FILE]...
was a three-stage pipeline all along: man ls produces the manual on stdout,
col -b cleans up its formatting, and head -n 8 keeps the first eight lines. Each
stage hands its stdout to the next. A simpler one: list data/ in long
form, keeping only the first few lines:
ls -l data | head -n 3
total 24
-rw-r--r-- 1 runner runner 2922 Jun 1 2023 README.md
drwxr-xr-x 2 runner runner 4096 Mar 1 2023 inputs
And one more, piping a file’s contents straight into head: the output of cat
becomes the input of head, no temporary file needed:
cat data/.dataset-notes | head -n 3
# .dataset-notes — a hidden file (its name starts with a dot, so `ls` skips it
# unless you ask with `ls -a`). Nothing secret; just a marker you can practise
# finding.
One caution worth repeating: only stdout flows through the pipe. A command’s
stderr is not piped onward; it leaks straight to your terminal unless you
redirect it (that is what the earlier 2> tricks are for). The pipe carries
results, not complaints.
Right now these pipelines are modest, because we only have NB1–3 commands to pipe.
That changes fast: the rest of Part II is nothing but tools built to sit in a
pipeline: grep to filter, cut/sort/uniq to reshape, awk to compute. The
pipe is what makes them worth having.
D. tee — save a copy without breaking the pipe#
Sometimes you want to keep an intermediate result and keep the pipeline flowing.
tee is the T-junction that does both: it writes its input to a file and passes
it through to the next command.
| -a | append to the file instead of overwriting it |
man tee-a, tee overwrites the file — the same clobber lesson as >ls data | tee scratch/listing.txt | head -n 2
README.md
inputs
The screen shows only the first two entries (because of head), but the full
listing was saved on the way past:
cat scratch/listing.txt
README.md
inputs
logs
results
scaling
trajectories
Operator reference#
These are shell syntax, not commands: there is no man >. They wire commands
and files together, and they are worth keeping in one place:
| > | send stdout to a file, overwriting it |
| >> | send stdout to a file, appending |
| < | take stdin from a file |
| | | pipe: one command's stdout becomes the next command's stdin |
| 2> | send stderr (errors) to a file |
| 2>&1 | send stderr wherever stdout is currently going |
| &> | send both stdout and stderr to a file |
| /dev/null | the void: discard whatever is redirected here |
Exercises#
From here on the exercises do more of the work; that is where fluency is built.
They ramp from mechanical warm-ups, through tasks on the real data/, to a
composite that wires the whole notebook together. As in Notebook 3, anything we
write goes into a fresh scratch/, never into data/.
Warm-up 1 (worked) — Save and read back#
Send a line of text into a scratch file with >, then read it back with cat.
hello from a redirected stream
✓ the line was saved to the file and read back unchanged
Warm-up 2 (your turn) — Append safely#
Create scratch/log.txt with a first line using >, then append two more
lines with >>, and cat the result. The file should end up holding exactly
three lines, in order. (Start with > so re-running the cell does not pile up
extra lines.)
line 1
line 2
line 3
✓ the file holds exactly three lines, line 1 first and line 3 last
Applied 3 (your turn) — Discard the noise#
Run ls on data and a file that does not exist, so it produces a valid
listing and an error. Send the error to /dev/null so only the listing
appears. (End the line with || true, as in the lesson.)
data:
README.md inputs logs results scaling trajectories
✓ the listing came through on stdout while the error stayed on stderr, which 2>/dev/null discarded
Applied 4 (worked) — A first pipeline#
Pipe a long listing of data/ into head to keep just the first five lines, then
re-run Notebook 1’s manual pipeline now that you can read it.
total 24
-rw-r--r-- 1 runner runner 2922 Jun 1 2023 README.md
drwxr-xr-x 2 runner runner 4096 Mar 1 2023 inputs
drwxr-xr-x 2 runner runner 4096 Feb 1 2023 logs
drwxr-xr-x 2 runner runner 4096 May 1 2023 results
LS(1) User Commands LS(1)
NAME
ls - list directory contents
SYNOPSIS
ls [OPTION]... [FILE]...
✓ the pipeline runs and head limits the listing to five lines
Applied 5 (your turn) — tee in the middle#
Pipe a plain listing of data/ through tee into scratch/listing.txt, then on
into head so you see only the top of it. Afterwards the file should hold the
full listing even though the screen showed only part.
README.md
inputs
✓ tee saved the full six-entry listing while head showed only the top
Composite — putting it together (capstone)#
Assemble a small report in scratch/report.txt, using everything in this notebook
in one sequence. Re-create the report with > at the top so reruns stay clean,
then append the rest with >>:
a header line built from
echoanddate(with>);a listing of
data/(with>>), suppressing any error with2>/dev/null;the first five lines of
data/trajectories/lj38-relaxed.xyz, sent throughtee -aso they are both appended to the report and previewed on screen;finally,
catthe finished report.
38
i = 28, E = -0.0621020546
Ar 4.5837264705 -2.5298506230 2.8419660895
Ar -5.0883572196 1.9357723701 2.4184523727
Ar -3.2418290672 -0.9829937359 -3.1034656658
=== data report — 2026-07-20 ===
Contents of data/:
README.md
inputs
logs
results
scaling
trajectories
First frame of lj38-relaxed.xyz:
38
i = 28, E = -0.0621020546
Ar 4.5837264705 -2.5298506230 2.8419660895
Ar -5.0883572196 1.9357723701 2.4184523727
Ar -3.2418290672 -0.9829937359 -3.1034656658
✓ the report holds its header, the data/ listing, and the trajectory's first frame
Optional stretch (terminal-only) — Compose your own#
No ✓ for this one. In the
live terminal,
build a three-stage pipeline from commands you already know (for example
ls -l data | head -n 4 | tail -n 2) and predict its output before you run
it. Then add a tee in the middle so you also save the intermediate result to a
file, and check the file matches what you expected. Predicting first, then
verifying, is how you build real confidence with pipes.
Outlook#
You now have the plumbing: the three streams, redirection to and from files, and
the pipe that snaps commands together. What is still missing is tools worth
piping, and that is the rest of Part II. Next (Notebook 5): globbing and brace
expansion, so a single command can act on many files at once; then grep, the
stream toolkit (cut/sort/uniq/wc/tr), and awk/sed: each a small,
sharp tool built to live inside a pipeline.
tee— copy what flows in to a file AND pass it straight through, so you can save a mid-pipeline result without breaking the pipeline.
See the full Compendium Scriptorum for every command met so far, and where to find it again.