5 Selecting files: globbing, brace expansion, and find

5 Selecting files: globbing, brace expansion, and find#

What this notebook is about#

Notebook 2 got you to a directory; Notebook 4 gave you pipes. But a real project has hundreds of files, and you are not going to name them one at a time. So this notebook answers a single question: how do you refer to many files at once, and act on the whole set?

There are three ways to select, and one way to act, and they share one idea:

glob: match files that already exist, here (*.xyz);
brace expansion: generate names and sequences, exist or not (run_{1..10});
find: search the whole tree by criteria (every .xyz, anywhere);
xargs: take any of those lists and run a command over all of it.

The files are the real .xyz trajectories and simulation logs from before; as ever, selecting them needs no physics. Let’s take the four in turn, simplest first.

A. Globbing — match files that are here#

A glob is a pattern the shell expands into a list of matching filenames. The pieces:

Globs — wildcards the shell expands into matching filenames (shell syntax, not a command).

*	any run of characters (including none)
?	exactly one character
[abc]	one character from the set
[1-5]	one character from the range
[!abc]	one character not in the set

* is the workhorse. Here it matches every .xyz one level down in data/:

ls data/*/*.xyz

data/results/pt-slab.xyz		 data/trajectories/lj38-relaxed.xyz
data/trajectories/lj38-optimization.xyz

The load-bearing idea is worth stopping on, because it explains half of bash’s surprises. The shell expands the glob before the command runs. ls never sees *.xyz; it sees the three real filenames the shell already substituted. You can watch that substitution directly with echo, which just prints whatever it is handed:

echo data/*/*.xyz

data/results/pt-slab.xyz data/trajectories/lj38-optimization.xyz data/trajectories/lj38-relaxed.xyz

The other wildcards need a batch of similarly-named files to show off, so let us make one in scratch/ (a stand-in for a project’s worth of numbered runs):

ls scratch/runs

run_1.log  run_1.xyz  run_10.log  run_2.log  run_2.xyz	run_3.log  run_3.xyz

? matches exactly one character, so run_?.log catches run_1 through run_9 but not run_10, which has two digits:

echo scratch/runs/run_?.log

scratch/runs/run_1.log scratch/runs/run_2.log scratch/runs/run_3.log

A bracket matches one character from a set or range:

ls scratch/runs/run_[1-2].log

scratch/runs/run_1.log	scratch/runs/run_2.log

Two gotchas worth carrying forward. First, a glob that matches nothing is, by bash’s default, passed through to the command literally: the pattern itself, unexpanded:

echo scratch/runs/*.dat

scratch/runs/*.dat

There is no .dat file, so *.dat arrived verbatim. (Bash has a nullglob option that makes a non-match expand to nothing instead; just know it exists.) Second, globs skip dotfiles: data/* quietly passed over the hidden .dataset-notes from Notebook 2:

echo data/*

data/README.md data/inputs data/logs data/results data/scaling data/trajectories

B. Brace expansion — generate names and sequences#

Globbing matches files that exist. Brace expansion is its complement: it generates strings whether or not any file exists, and the shell does it even earlier than globbing.

Brace expansion — generate lists and sequences of strings (shell syntax). Runs before globbing; the files need not exist.

{a,b,c}	each item in turn (no spaces inside!)
{1..10}	a numeric range
{1..10..2}	a range with a step
pre{a,b}post	a shared prefix/suffix is distributed over each item
file{,.bak}	the empty item gives `file` and `file.bak`: the backup idiom

Watch the generation with echo:

echo run_{1..5}

run_1 run_2 run_3 run_4 run_5

echo frame_{0..10..2}.xyz

frame_0.xyz frame_2.xyz frame_4.xyz frame_6.xyz frame_8.xyz frame_10.xyz

Because the strings are generated regardless of what exists, braces are how you create a structured set in one stroke. mkdir -p run_{1..5} makes five directories at once:

mkdir -p scratch/run_{1..5}

ls scratch

run_1  run_2  run_3  run_4  run_5

And the file{,.bak} idiom expands cp config.txt{,.bak} into cp config.txt config.txt.bak: an instant backup in nine keystrokes:

cp scratch/config.txt{,.bak}

ls scratch

config.txt  config.txt.bak

C. `find` — search the whole tree by criteria#

Globs and braces work on names in one directory. find makes the leap from “names here” to “criteria anywhere in the tree”: walk a directory and everything under it, keep the files that match a set of tests, and (optionally) run a command on each.

find — search a directory tree for files matching criteria — name, type, size, time — and optionally act on each one.

-name "PAT" / -iname	match the filename against a quoted glob (`-iname` ignores case)
-type f / -type d	match only files / only directories
-maxdepth N	descend at most N levels deep
-size +Nk / -mtime -N	by size (larger than N kB) or age (changed within N days)
-exec CMD {} +	run CMD on the matches — `{}` is each path, `+` batches them into few calls
-print / -delete	print the matches (the safe default) / delete them (dangerous — see the admonition)

more: man find — its options are many; you consult it, you do not memorise it

Watch out: quote the -name glob (find data -name "*.xyz") or the shell expands it before find ever sees it; and -delete is rm at scale — read the admonition first

The shape is find <where> <tests> <action>. Start simple: every .xyz underneath data/, wherever it lives:

find data -name "*.xyz"

data/trajectories/lj38-optimization.xyz
data/trajectories/lj38-relaxed.xyz
data/results/pt-slab.xyz

It reached into both trajectories/ and results/, something no single glob can do. Quote the pattern ("*.xyz"): that is the payoff of §A. If you leave it bare, the shell expands *.xyz before find ever runs, and find gets the wrong thing. Quoting hands the pattern to find intact.

Tests combine. Only directories:

find data -type d

data
data/inputs
data/logs
data/scaling
data/trajectories
data/results

Keep it shallow with -maxdepth, and select by size — here, files bigger than 50 kilobytes, which finds the one large trajectory:

find data -size +50k

data/trajectories/lj38-optimization.xyz

Tests can be negated with ! and combined with -o (or). This finds every .xyz that is not under results/:

find data -name "*.xyz" ! -path "*/results/*"

data/trajectories/lj38-optimization.xyz
data/trajectories/lj38-relaxed.xyz

⚠ find can delete at scale — dry-run first, always

find has a -delete action and an -exec rm action, and they are the Notebook 3 rm footgun multiplied by every file in the tree. One wrong test (a -name that is broader than you meant, a stray -o) and an entire subtree is gone, with no undo and no trash. The habit that saves you is simple and non-negotiable: run the find with -print first and read the list, every time. Only once you have seen exactly what it matches do you swap -print for -delete. Never pipe a find you have not eyeballed into anything that removes files.

D. `xargs` — act on a list#

find selects files; xargs acts on them. It reads a list of items on standard input and turns them into arguments for a command, which means it is, at heart, just a pipe (Notebook 4): a list flows in, command lines come out.

xargs — read a list of items on standard input and turn them into arguments for a command — the bridge from a pipeline that produces names to a command that acts on them.

-0	items are NUL-separated, not whitespace — pair with `find -print0` to survive spaces and newlines in names
-n N	use at most N items per command line
-I {}	place each item where `{}` appears, instead of appending it at the end

more: man xargs

Watch out: filenames with spaces or newlines break the default whitespace splitting — use find -print0 | xargs -0. (It also has -P to run jobs in parallel; we return to that in Part IV.)

The classic pairing pulls the first line (the atom count) out of every trajectory find can reach:

find data -name "*.xyz" -print0 | xargs -0 head -n 1

==> data/trajectories/lj38-optimization.xyz <==
      38
==> data/trajectories/lj38-relaxed.xyz <==
      38
==> data/results/pt-slab.xyz <==
180

Two details carry the safety lesson. find … -print0 separates names with an invisible NUL character instead of whitespace, and xargs -0 reads them the same way, so a filename with a space or newline in it cannot be split in two and mangled. Make -print0 | xargs -0 your default pairing.

find also has its own built-in way to act, -exec, which needs no pipe at all. The {} stands for each match and the trailing + batches them efficiently:

find data -name "*.xyz" -exec head -n 1 {} +

==> data/trajectories/lj38-optimization.xyz <==
      38
==> data/trajectories/lj38-relaxed.xyz <==
      38
==> data/results/pt-slab.xyz <==
180

Same result, two routes: lead with xargs when you want a general “list → arguments” tool that composes with any producer, and reach for -exec when the list is coming from find anyway. (One more thing xargs can do, for later: it will run those command lines in parallel with -P. We come back to that when we talk about speed in Part IV.)

Exercises#

A fuller set this time: selection is a theme worth practising from several angles. Reads over data/ are fine; anything that creates or deletes works in a fresh scratch/, set up at the top of each exercise so reruns are identical.

Warm-up 1 (worked) — Globs#

Match files with globs and watch a non-match pass through literally.

scratch/runs/sweep_1.xyz  scratch/runs/sweep_2.xyz  scratch/runs/sweep_3.xyz
scratch/runs/sweep_1.log  scratch/runs/sweep_3.log  scratch/runs/sweep_5.log
scratch/runs/sweep_2.log  scratch/runs/sweep_4.log
scratch/runs/*.dat

✓ the .xyz glob matched 3, sweep_?.log matched 5, and the non-match passed through literally

Warm-up 2 (your turn) — Brace expansion#

In scratch/, make five run directories run_1 … run_5 in one mkdir -p, and make a backup of a file with the file{,.bak} idiom.

input.txt  input.txt.bak  run_1  run_2	run_3  run_4  run_5

✓ five run_ directories exist and the .bak backup was made

Applied 1 (your turn) — find by criteria#

Using find on data/ (read-only): find every .inp input file (quote the pattern!), and separately every directory. Then list only what is directly inside data/ with -maxdepth 1.

data/inputs/geo-opt.inp
data/inputs/production-md.inp
data
data/inputs
data/logs
data/scaling
data/trajectories
data/results
data
data/README.md
data/.dataset-notes
data/inputs
data/logs
data/scaling
data/trajectories
data/results

✓ find matched both .inp inputs and all six directories

Applied 2 (worked) — find | xargs#

Pull the first line out of every trajectory in data/, two ways: piped through xargs, and with find’s own -exec.

==> data/trajectories/lj38-optimization.xyz <==
      38
==> data/trajectories/lj38-relaxed.xyz <==
      38
==> data/results/pt-slab.xyz <==
180

==> data/trajectories/lj38-optimization.xyz <==
      38
==> data/trajectories/lj38-relaxed.xyz <==
      38
==> data/results/pt-slab.xyz <==
180

✓ all three trajectories were found and their first lines pulled

Composite — putting it together (capstone)#

Build a small sorted archive in scratch/, exercising the whole notebook at once. Make three brace-generated bins scratch/sorted/{R,S,M}, then use find + xargs to copy every .xyz from data/ into the R bin, and verify with find.

scratch/sorted/R/pt-slab.xyz
scratch/sorted/R/lj38-optimization.xyz
scratch/sorted/R/lj38-relaxed.xyz

✓ the three brace-made bins exist and all three .xyz were copied into R/

Optional stretch (your turn) — Boolean find#

Compose a single find with combined tests: every .xyz under data/ that is not in results/ and is larger than 1 kilobyte, then count them with xargs. (Aside: xargs could run the action in parallel with -P, a thread we pick up in Part IV.)

-rw-r--r-- 1 runner runner 155K Apr  1  2023 data/trajectories/lj38-optimization.xyz
-rw-r--r-- 1 runner runner 2.5K Apr  1  2023 data/trajectories/lj38-relaxed.xyz

✓ the boolean find selected exactly the two large trajectories outside results/

Outlook#

You can now select files at scale (by pattern, by generation, by searching the tree) and act on the whole set at once. Next (Notebook 6): instead of finding files, you will find matching text inside them with grep, and meet regular expressions: the pattern language that sed, and the Vim search-and-replace from Notebook 9, all share.

New in the Compendium

find — search a directory tree for files matching criteria — name, type, size, time — and optionally act on each one.
xargs — read a list of items on standard input and turn them into arguments for a command — the bridge from a pipeline that produces names to a command that acts on them.

See the full Compendium Scriptorum for every command met so far, and where to find it again.