3 Looking at and handling files

3 Looking at and handling files#

What this notebook is about#

Notebook 2 got you to a file: you can find any file in the tree. Now you do the two things you actually came for: look inside a file, and move files around.

As before, the playground is the course’s real data/ folder: .xyz trajectories and simulation inputs from the Molecular and Materials Modelling course. You do not need to understand any of it. The only thing that matters here is that these are text files, and the skills (viewing them and organising them) are the same whether a file holds atomic coordinates or a shopping list.

One ground rule for this notebook: we treat data/ as read-only. Anything we create, copy, or delete happens in a throwaway scratch/ directory, so there is no way to harm the real files. (This is also just good practice.)

A. Looking inside files#

`file` — what am I even looking at?#

Before you open something, it is worth knowing what it is. file peeks at a file’s contents and tells you its type.

file — identify what kind of file something is, before you open it.

more: man file

Watch out: check first — cat-ing a binary file dumps control characters and can garble your terminal; file tells you whether it is safe text

file data/trajectories/lj38-relaxed.xyz

data/trajectories/lj38-relaxed.xyz: ASCII text

Plain text, safe to read. That check matters more than it looks: opening a binary file as if it were text can spray control characters across your terminal and garble it. file is how you avoid that.

`cat` — print the whole thing#

The bluntest way to see a file is cat, which dumps its entire contents to the screen.

cat — print a file's contents straight to the screen.

-n	number every line

more: man cat

Watch out: cat prints the WHOLE file — on a huge one it floods your screen (reach for head/tail/less); on a binary it garbles the terminal (run file first)

On a short file that is exactly what you want. Here is the hidden notes file from the data folder:

cat data/.dataset-notes

# .dataset-notes — a hidden file (its name starts with a dot, so `ls` skips it
# unless you ask with `ls -a`). Nothing secret; just a marker you can practise
# finding.
#
# These files are real Molecular & Materials Modelling (ETH Zürich) outputs,
# republished as a navigation playground. You never need to understand the
# physics to practise the shell on them.
provenance: MMM FS2023
scrubbed: usernames and cluster hostnames removed (none were present)

Add -n to number the lines:

cat -n data/.dataset-notes

# .dataset-notes — a hidden file (its name starts with a dot, so `ls` skips it
# unless you ask with `ls -a`). Nothing secret; just a marker you can practise
# finding.
#
# These files are real Molecular & Materials Modelling (ETH Zürich) outputs,
# republished as a navigation playground. You never need to understand the
# physics to practise the shell on them.
provenance: MMM FS2023
scrubbed: usernames and cluster hostnames removed (none were present)

But heed the gotcha on the card. The optimisation trajectory in data/ is 2,520 lines long; cat-ing it would bury your screen in coordinates. When a file is big, you do not want all of it; you want a piece of it. That is the next two commands.

`head` and `tail` — just the ends#

head shows the first few lines of a file; tail shows the last few. They are how you sample a big file without drowning in it.

head — show the first lines of a file.

-n N	show the first N lines (default 10)
-c N	show the first N bytes instead of lines

more: man head

tail — show the last lines of a file.

-n N	show the last N lines (default 10)
-c N	show the last N bytes instead of lines
-f	follow — keep printing new lines as the file grows (watch a running job's log live); stop with Ctrl+C

more: man tail

Watch out: tail -f never returns on its own — it waits for more; press Ctrl+C to get your prompt back

A one-line motivation, then the release valve: in an .xyz trajectory the first line is the atom count and the second carries the step and energy. But you do not need to know that. The point is simply that head pulls the top:

head -n 3 data/trajectories/lj38-relaxed.xyz

      38
 i =      28, E =       -0.0621020546
 Ar         4.5837264705       -2.5298506230        2.8419660895

…and tail pulls the bottom, here the last three lines of the 2,520-line trajectory, fetched instantly without reading the rest:

tail -n 3 data/trajectories/lj38-optimization.xyz

 Ar         8.2684872923       10.6869340322        9.8193483259
 Ar        10.3806592580        9.2923533596       12.5676466468
 Ar         9.5922722829        7.1863024355        9.5820081406

One flag on tail is worth singling out: tail -f follows a file, printing new lines as they are added. This is how you watch a running job’s log live, an everyday move on a cluster. It is interactive (it keeps waiting), so it belongs in a real terminal, not a static cell:

Practise this in your terminal

Open the live terminal, start writing to a file in one place and tail -f it — watch the new lines appear as they are written. Press Ctrl+C to stop following.

`less` — page through a big file#

To actually read a long file (not just its ends), use the pager, less. It shows one screen at a time and lets you scroll and search.

less — page through a file one screen at a time — the pager. (Its “flags” are keys you press while it is open.)

Space / b	page down / page up
/pattern → n / N	search forward, then jump to the next / previous match
g / G	jump to the top / the bottom
q	quit

more: man less

Watch out: man shows its pages through less, so the same keys work in both — q quits each, / searches each. It is interactive: practise it in the terminal

less is interactive (there is nothing to print on a page, only keys to press), so this one, too, is for your terminal. There is a payoff for a habit you already have: man displays its pages through less, so every key you learn here works in man as well. q quits both; / searches both.

Practise this in your terminal

In the live terminal, run less data/inputs/geo-opt.inp. Page with the spacebar, search with /csvr (or any word), jump around with g and G, and quit with q. Then try man ls and notice the keys are identical.

B. Handling files#

Now the other half: making, copying, renaming, and removing files and folders. Everything here happens in a fresh scratch/ directory so the real data stays untouched.

`touch` and `mkdir` — make files and folders#

touch makes an empty file (or updates a file’s timestamp); mkdir makes a directory.

touch — create an empty file, or update an existing file's timestamp.

more: man touch

mkdir — make a new directory.

-p	create parent directories as needed, and do not error if it already exists

more: man mkdir

Watch out: making a nested path like a/b/c in one go needs -p; without it, mkdir fails unless every parent already exists

touch scratch/notes.txt

touch printed nothing; like most of these commands, silence means success. mkdir with -p makes a whole nested path at once (and never complains if it already exists):

mkdir -p scratch/results/raw

Let’s see what we built:

ls -R scratch

scratch:
notes.txt  results
scratch/results:
raw
scratch/results/raw:

`cp` — copy#

cp copies a file. To copy a directory, you must add -r.

cp — copy a file (or, with -r, a directory).

-r	recursive — copy a directory and everything inside it
-i	prompt before overwriting an existing file
-v	verbose — print each file as it is copied

more: man cp

Watch out: copying a directory needs -r; and cp overwrites the destination SILENTLY unless you add -i

cp scratch/notes.txt scratch/notes-backup.txt

ls scratch

notes-backup.txt  notes.txt  results

`mv` — move and rename#

mv moves a file somewhere else, and renaming is the same operation, just moving a file to a new name in the same directory.

mv — move a file — or rename it (same command: renaming is just moving within a directory).

-i	prompt before overwriting an existing file
-v	verbose — print what was moved

more: man mv

Watch out: like cp, mv overwrites the destination silently — add -i if there is any chance it already exists

mv scratch/notes-backup.txt scratch/notes-v2.txt

ls scratch

notes-v2.txt  notes.txt  results

The backup is gone and notes-v2.txt is in its place: that was a rename.

`rm` and `rmdir` — remove#

rm deletes a file; rmdir deletes an empty directory.

rm — remove (delete) a file. Permanently.

-r	recursive — delete a directory and everything in it
-i	prompt before each removal (a good habit)
-f	force — never prompt, ignore missing files (dangerous; see the admonition)

more: man rm

Watch out: there is no undo and no trash. rm -rf is the classic footgun — read the admonition in the notebook before you lean on it

rmdir — remove an empty directory — and only an empty one.

more: man rmdir

Watch out: it refuses to delete a non-empty directory — that narrowness is the safety; use it when you mean “this should be empty”

rm scratch/notes-v2.txt

ls scratch

notes.txt  results

That file is now gone. A word about what “gone” really means:

⚠ rm has no undo, and no trash

This is the most consequential warning in the course so far, so read it once, properly. When rm deletes a file, it is gone: there is no Recycle Bin, no Trash, no “are you sure?”. rm -rf somedir will silently erase an entire directory tree, and rm -rf * in the wrong directory has ruined many a day. Three habits keep you safe:

Look before you leap. Before rm-ing with a wildcard, run the same pattern through ls first: ls *.tmp shows you exactly what rm *.tmp will destroy.
Use rm -i when it matters. It prompts before each deletion; the half-second pause has saved many files.
Reach for rmdir when you mean “empty”. It refuses to delete a non-empty directory, so it cannot take anything down with it by surprise.

rm is not to be feared; it is to be respected, like any sharp tool.

Exercises#

The pattern is unchanged: a task, a place for your answer, an automatic ✓. Reads come from data/; anything that creates or deletes works in scratch/, set up fresh each time.

Exercise 1 (worked) — Identify, then peek#

Find out what data/inputs/geo-opt.inp is with file, then show its first few and last few lines with head and tail.

data/inputs/geo-opt.inp: ASCII text
&GLOBAL                       ! sets general information about the calculation
   PRINT_LEVEL LOW            ! prints only essential output
   PROJECT optimization       ! sets name of project
   RUN_TYPE GEO_OPT           ! type of calculation is a geometry optimization
      &END
  &END
 &END SUBSYS
&END FORCE_EVAL

✓ file reports a text type, and head and tail both produce output

Exercise 2 (your turn) — First and last of a trajectory#

Show the first 5 lines and the last 5 lines of the big trajectory data/trajectories/lj38-optimization.xyz, without reading the 2,510 lines in between. Reach for head -n 5 and tail -n 5.

      38
 i =        1, E =        -0.0231086608
 Ar        16.8773225351        7.2595905272       13.9246835670
 Ar         5.1445774260       12.1651706949       12.9335732081
 Ar         5.2735834035        7.6991362952        4.8232906858
 Ar        10.9581730448        9.8649271971        7.3611169007
 Ar        13.0734311384        8.4718065837       10.1062562464
 Ar         8.2684872923       10.6869340322        9.8193483259
 Ar        10.3806592580        9.2923533596       12.5676466468
 Ar         9.5922722829        7.1863024355        9.5820081406

✓ head -n 5 and tail -n 5 each return exactly five lines

Exercise 3 (your turn) — Organize a results folder#

In scratch/: make a directory inputs/ (use mkdir -p), copy data/inputs/geo-opt.inp into it, rename the copy to run-01.inp with mv, and list the result with ls. The original in data/ must stay put.

run-01.inp

✓ the copy was renamed to run-01.inp, and the original data file is untouched

Exercise 4 (your turn) — Safe removal#

In scratch/: touch two files, keep.txt and delete-me.txt, then remove only delete-me.txt. Afterwards keep.txt must still be there; removing one thing should never take its neighbours with it.

keep.txt

✓ delete-me.txt is gone and keep.txt is untouched

Exercise 5 (terminal-only) — Paging and following#

No ✓ for this one: it is interactive. In the live terminal: open less data/results/sn2-neb.ener, search it with /, and quit with q. Then pick a file and tail -f it while you append to it from elsewhere, and watch the new lines arrive live. These two habits, paging and following, are everyday moves once you are working on real runs.

Outlook#

You can now see what is inside a file and move files around the tree, which closes out the orientation part of the course. Part II is where it turns from handling files to mining them. Instead of reading a file by eye, you will extract from it: feed cat or head into tools that filter and reshape text, starting with redirection and pipes, then grep, the stream toolkit, and awk/sed. Everything you have built (moving through the tree, peeking at files) is the ground that stands on.

New in the Compendium

file — identify what kind of file something is, before you open it.
cat — print a file's contents straight to the screen.
head — show the first lines of a file.
tail — show the last lines of a file.
less — page through a file one screen at a time — the pager. (Its “flags” are keys you press while it is open.)
touch — create an empty file, or update an existing file's timestamp.
mkdir — make a new directory.
cp — copy a file (or, with -r, a directory).
mv — move a file — or rename it (same command: renaming is just moving within a directory).
rm — remove (delete) a file. Permanently.
rmdir — remove an empty directory — and only an empty one.

See the full Compendium Scriptorum for every command met so far, and where to find it again.