15 Remote work and moving data#
What this notebook is about#
Everything so far you could do on the machine in front of you. The cluster changes
one thing: you are not sitting at it. It is a computer somewhere else (in a
machine room, reached over the network), and that single fact creates three needs.
You have to get on and stay on it (ssh, tmux). You have to move your data
across to it and your results back (tar, scp, rsync). And because the real
work takes hours or days, you have to launch it, disconnect, and come back to
find it still running and watchable (&, nohup, ps, top, kill). One frame,
a machine you’re not at, holds this whole wide toolkit together.
Which cells run here, and which are “type it yourself”
There is no second machine inside this page, so some of these tools cannot run in the grey cells. We label every command:
A live grey cell with output below it runs for real, right here: the transfer and process mechanics (
tar,rsynclocal-to-local,du,ps,ssh-keygen, …) are all genuine.A plain code block tagged
[your terminal]needs a real remote machine or an interactive screen (sshto a cluster,scp,tmux,top). Type those in the Practice here terminal, or (for the connect-and-copy ones) against a real cluster like ETH’s Euler.
The good news: the mechanics of moving data are identical whether the
destination is a folder next door or a supercomputer. rsync run locally teaches
the real thing; you just add host: to the path when the destination is remote.
A. Reaching the machine — ssh#
ssh (“secure shell”) is the front door to every cluster. Point it at a
machine and it opens a shell there, encrypted end to end, exactly as if you had sat
down at its keyboard:
| user@host | log in as user on host and get an interactive remote shell |
| host CMD | run ONE command on the remote and return (e.g. ssh euler squeue) |
| -i keyfile | authenticate with a specific private key |
| -p N | connect on port N (the default is 22) |
man ssh; host aliases live in ~/.ssh/config (Host euler … lets you just type ssh euler)chmod 600 ~/.ssh/id_* (Notebook 11). Keys beat passwords: generate with ssh-keygen, install on the remote with ssh-copy-idIt comes in two shapes: an interactive session, or a single remote command:
# [your terminal]
ssh you@euler.ethz.ch # opens an interactive shell ON Euler
ssh euler squeue # runs ONE command there and returns its output
Keys, not passwords#
Typing your password on every connection is both tedious and less secure than the standard alternative: a key pair. You generate two matching files (a private key you keep, and a public key you install on the cluster) and ssh proves your identity with them automatically. Generating the pair is real, so run it here:
ssh-keygen -t ed25519 -f scratch/demo_key -N "" -C "you@laptop"
Generating public/private ed25519 key pair.
Your identification has been saved in scratch/demo_key
Your public key has been saved in scratch/demo_key.pub
The key fingerprint is:
SHA256:QXL3+vH7R8VsYa0a9js1au89mziX86qoCNCauGF1SCw you@laptop
The key's randomart image is:
+--[ED25519 256]--+
| . o . .|
| . + . . o.|
| E o . . .+.|
| o.. . .o . =|
| .o.. S ...+ ..|
| ..+. ..o..o|
|o.o . . o+o|
|.o . . . ++*+|
|. . ... ooBOX|
+----[SHA256]-----+
That made two files. Look at their permissions, and notice ssh-keygen has already done the Notebook-11 chmod for you:
ls -l scratch/demo_key scratch/demo_key.pub
-rw------- 1 runner runner 399 Jul 20 06:09 scratch/demo_key
-rw-r--r-- 1 runner runner 92 Jul 20 06:09 scratch/demo_key.pub
The private key is -rw------- (mode 600), readable only by you. That is not
optional:
⚠ ssh refuses a private key others can read
If your private key’s file is readable by group or others, ssh ignores it and
falls back to asking for a password: the classic “but I set up my key!” confusion.
The fix is the Notebook-11 one: chmod 600 ~/.ssh/id_ed25519. Keys are private;
the filesystem has to agree.
You install the public half on the cluster with ssh-copy-id you@euler, and then
a short ~/.ssh/config turns a long connection string into a one-word alias:
# [your terminal]
# ~/.ssh/config
Host euler
HostName euler.ethz.ch
User your_username
IdentityFile ~/.ssh/id_ed25519
# now this is all you type:
ssh euler
B. Moving data — tar, then rsync#
Before you transfer a result directory, you usually bundle and compress it: one
file moves faster and more reliably than ten thousand. That is tar (+ gzip):
| -czf a.tar.gz dir/ | CREATE a gzip-compressed archive of dir/ (c=create, z=gzip, f=file) |
| -xzf a.tar.gz | EXTRACT a gzip archive |
| -tzf a.tar.gz | LIST an archive's contents without extracting |
| -C dir | change to dir first — archive from, or extract into, a chosen directory |
man tar; the create mnemonic is czf — "compress, zip, file"f comes LAST among the bundled flags because the filename follows it (-czf name); a gzip-compressed tar is conventionally .tar.gz (or .tgz)| file | compress to file.gz, removing the original |
| -d / gunzip | decompress (gunzip file.gz is the same as gzip -d) |
| -k | keep the original alongside the compressed copy |
| -9 / -1 | best compression (slow) / fastest (less compression) |
man gzip; for a whole directory, tar -czf first and you get one compressed bundlegzip file REPLACES the file with file.gz (add -k to keep both); gzip handles ONE file — tar is what bundles manyCreate an archive, list it without unpacking, and extract it again, all real:
tar -czf scratch/run.tar.gz -C scratch run
tar -tzf scratch/run.tar.gz
run/
run/result.txt
run/logs/
run/logs/run.log
mkdir -p scratch/restored && tar -xzf scratch/run.tar.gz -C scratch/restored && find scratch/restored -type f
scratch/restored/run/result.txt
scratch/restored/run/logs/run.log
The archive round-tripped: bundled with c, inspected with t, unpacked with x.
(gzip on its own compresses a single file: gzip big.log → big.log.gz; tar
is what bundles many.)
scp — the one-shot copy#
The simplest remote transfer is scp, which copies over ssh just like cp
copies locally:
| file host:path | copy a local file UP to the remote |
| host:path . | copy a remote file DOWN to here |
| -r | copy a directory and everything in it |
man scp; for anything beyond a one-off, prefer rsync — it resumes and skips unchanged fileshost:path, with a COLON — forget the colon and scp just makes a local copy named host# [your terminal]
scp run.tar.gz euler:/cluster/scratch/you/ # push a file up
scp euler:results/summary.txt . # pull a file down
scp -r mydir euler:~/ # a whole directory
rsync — the workhorse#
For anything you will transfer more than once (a results directory you re-sync as a
job produces more), rsync is the right tool. It copies only what has
changed, so the second sync is near-instant. Crucially, its local-to-local form is
the same tool you use for the cluster, so we practise it for real:
| -a | archive mode: recurse and preserve permissions, times, and symlinks (the everyday default) |
| -v / --progress | verbose / show a progress bar on large transfers |
| -z | compress data in transit (helps over a slow link) |
| --dry-run | show what WOULD transfer, changing NOTHING — run this first |
| --delete | delete files on the destination that are gone from the source (dangerous — see the admonition) |
man rsync; the SAME command syncs to a remote by writing host:path for either sidersync -a src/ dst copies the CONTENTS of src into dst, while rsync -a src dst copies the src DIRECTORY into dst. And --delete can wipe the destination — --dry-run first, every timersync -av scratch/src/ scratch/dst/
sending incremental file list
a.txt
b.txt
sent 187 bytes received 54 bytes 482.00 bytes/sec
total size is 4 speedup is 0.02
Now change one file and re-sync: rsync transfers only the changed file, not the whole set:
printf 'a CHANGED\n' > scratch/src/a.txt; rsync -av scratch/src/ scratch/dst/
sending incremental file list
a.txt
sent 150 bytes received 35 bytes 370.00 bytes/sec
total size is 12 speedup is 0.06
Only a.txt moved the second time. And preview before you sync with
--dry-run, which lists what would happen and touches nothing:
printf 'c\n' > scratch/src/c.txt; rsync -av --dry-run scratch/src/ scratch/dst/
sending incremental file list
c.txt
sent 112 bytes received 19 bytes 262.00 bytes/sec
total size is 14 speedup is 0.11 (DRY RUN)
ls scratch/dst
a.txt b.txt
The dry run named c.txt as pending, but scratch/dst still has only a and b;
nothing changed. One footgun deserves a hard stop:
⚠ The trailing slash, and --delete
Trailing slash: rsync -a src/ dst copies the contents of src into dst;
rsync -a src dst copies the directory src into dst (giving dst/src/…).
Mixing these up is the most common rsync surprise: be deliberate about the /.
--delete: this makes the destination an exact mirror by deleting files there
that are gone from the source. It is exactly what you want for a true mirror, and
a disaster if you point it at the wrong directory. So: --dry-run first, every
time, and read the list before you run it for real.
C. Staying on — tmux and job control#
Here is the problem the cluster forces on you. You start a six-hour job over ssh,
close your laptop to go home, and your connection drops, the shell receives a
hang-up signal (SIGHUP), and your job dies with it. Two tools solve this.
tmux — a session that outlives the connection#
tmux runs a terminal on the remote machine itself that keeps going when you
detach. You start work inside it, press Ctrl-b d to detach (the work keeps
running), close your laptop, and later ssh back in and tmux attach to find
everything exactly as you left it.
| tmux | start a new session |
| Ctrl-b d | DETACH — leave it running and drop back to your normal shell |
| tmux attach / tmux a | re-attach to the running session after you reconnect |
| tmux ls | list running sessions |
man tmux; screen is the older equivalent. The prefix Ctrl-b precedes every tmux commandSIGHUP and kills your running commands — tmux is how a long job survives you closing the laptop# [your terminal]
ssh euler
tmux # start a persistent session
./long_analysis.sh # launch your work inside it
# ... press Ctrl-b then d to DETACH; now it's safe to disconnect ...
exit # log out of ssh; the job keeps running on Euler
# ... hours later, from anywhere ...
ssh euler
tmux attach # back exactly where you left off
Job control — &, jobs, nohup#
For a single command, lighter tools suffice. End a command with & to run it in
the background, freeing your prompt; jobs lists what is backgrounded. Run
it for real: start a long sleep, see it listed, and we will stop it in §D:
sleep 300 &
echo "backgrounded as PID $!"
[1] 4651
backgrounded as PID 4651
jobs
[1]+ Running sleep 300 &
| cmd & | run cmd in the background; the prompt returns at once |
| jobs | list the shell's background jobs, with their numbers |
| Ctrl-z | SUSPEND the foreground command (pauses it, hands back the prompt) |
| bg / fg | resume a suspended job in the background / foreground |
| $! / %1 | the PID of the last background command / a job by its number |
And nohup makes a command deaf to that hang-up signal, so it survives logout
even without tmux:
| nohup CMD & | run CMD in the background, ignoring the SIGHUP sent at logout |
man nohup; output it would have printed goes to nohup.out unless you redirect ittmux is usually nicer; nohup … & is the quick "launch it and let it run" for a single commandnohup sleep 300 > nohup.out 2>&1 &
echo "nohup PID $! — still running after logout; output in nohup.out"
[2] 4652
nohup PID 4652 — still running after logout; output in nohup.out
(nohup parks any output the command would have printed into a file called
nohup.out. For real work you usually redirect it somewhere meaningful instead.)
D. Watching and resources#
Now to watch those processes and stop them. ps takes a snapshot of what is
running: pair it with grep to find one. Both sleeps we backgrounded are alive:
| aux | ALL processes for all users, with CPU/memory (the common BSD-style form) |
| -ef | the same idea in System-V style; pipe into grep to find one |
| -p PID | show just one process by PID |
man ps; for a LIVE, updating view rather than a snapshot, use top / htopps is a still photo, not a live feed — re-run it (or use top) to see change. Find your process with ps aux | grep NAMEps aux | grep "[s]leep 300"
runner 4651 0.0 0.0 6124 1976 pts/0 S+ 06:09 0:00 sleep 300
runner 4652 0.0 0.0 6124 1972 pts/0 S+ 06:09 0:00 sleep 300
For a live, continuously updating view there is top (and the friendlier
htop), interactive, so it belongs in a real terminal:
| top | start the live view (press q to quit) |
| htop | a friendlier colour version, if installed — scrollable, with click-to-kill |
man top; it is interactive, so practise it in a real terminaltop takes over the screen and never returns on its own — press q to quit (like less and man)# [your terminal]
top # live process table; press 'q' to quit
htop # the nicer colour version, if installed
To stop a process, kill sends it a signal by PID. Plain kill asks politely
(it can clean up first); kill -9 is the no-mercy hammer. Let’s stop the
background sleeps we started, politely:
| kill PID | politely ask it to terminate (SIGTERM — it can clean up first) |
| kill -9 PID | force-kill (SIGKILL — immediate, no cleanup; the last resort) |
| kill %1 | refer to a background job by its jobs number instead of its PID |
man kill; get the PID from ps or jobskill first; kill -9 cannot be caught, so the process gets NO chance to flush files or shut down cleanly — reach for it only when something is truly stuckkill %1 %2 2>/dev/null; wait 2>/dev/null; jobs
Both jobs are gone. (When you only have a PID, kill <PID>; reach for kill -9
only when a process ignores a polite kill.)
watch — re-run something to see it change#
watch re-runs a command every couple of seconds in place: perfect for
keeping an eye on a job queue. It is interactive, so it too is a terminal tool:
| watch CMD | re-run CMD every 2 seconds (e.g. watch squeue to watch a job queue) |
| -n N | set the interval to N seconds |
| -d | highlight what changed since last time |
man watch; quit with Ctrl-CCtrl-C to stop) — practise it in a real terminal# [your terminal]
watch squeue # refresh the SLURM queue every 2s (Notebook 16)
watch -n 5 'ls -l results/ | tail' # watch results appear, every 5s
Disk awareness — du and df#
Clusters give you a quota, and jobs that fill it fail in confusing ways. Two
commands keep you ahead of it. du measures what your directories use:
| -s | summary: one total per argument, not every file within |
| -h | human-readable sizes (K, M, G) |
| -sh * | the everyday combo: a readable total for each item here |
man du; pair with sort -h to rank by size (du -sh * | sort -h)du measures what a directory's CONTENTS use (so it can be slow on a huge tree); df measures the disk/quota as a wholedu -sh scratch/big scratch/small
200K scratch/big
8.0K scratch/small
…and df measures the filesystem (or quota) as a whole:
| -h | human-readable sizes |
| df -h . | show the filesystem holding the current directory |
man df; on a cluster a per-user quota tool (e.g. lfs quota, quota) reports your real limitdf is about the whole filesystem; to find which of YOUR directories is the culprit, that is dudf -h "$ROOT" | tail -n 1
/dev/root 72G 57G 15G 80% /
(du finds which directory is heavy; df tells you how full the disk is.)
E. Symlinks — ln -s#
One last cluster staple. Your big working space (often /scratch/$USER) is somewhere
inconvenient, so you make a symbolic link (a lightweight signpost) to reach it
by a short name from where you work:
| -s target link | create a SYMBOLIC link named link pointing at target |
| -sf | force — replace an existing link of that name |
man ln; ls -l shows a symlink as link -> target (Notebooks 2–3)-s (symbolic). The classic cluster move is ln -s /scratch/$USER scratch, so a short local name points at your big scratch spaceln -s realdir scratch/link-to-realdir; ls -l scratch/link-to-realdir
lrwxrwxrwx 1 runner runner 7 Jul 20 06:09 scratch/link-to-realdir -> realdir
The -> in ls -l (Notebooks 2–3) shows it is a pointer, not a copy. Follow it and
you reach the real thing:
cat scratch/link-to-realdir/data.txt
the real file
Exercises#
The runnable ones work in a fresh scratch/; data/ stays read-only, and every
backgrounded process is cleaned up. The remote ones are yours to do in a terminal
(the Practice here box, or against a real cluster).
Warm-up 1 (worked) — Archive round-trip#
tar -czf a directory, list it with -tzf, extract it elsewhere with -xzf, and
confirm the contents survived.
--- contents ---
results/
results/summary.txt
results/data.csv
--- extracted ---
scratch/out/results/data.csv
scratch/out/results/summary.txt
✓ the archive was created, listed, and extracted with its contents intact
Warm-up 2 (your turn) — Local rsync#
rsync -av a source directory into a destination; change one file and re-sync (note
only it moves); then --dry-run a new file and confirm the destination did not
change.
sending incremental file list
one.txt
sent 150 bytes received 35 bytes 370.00 bytes/sec
total size is 8 speedup is 0.04
sending incremental file list
three.txt
sent 124 bytes received 19 bytes 286.00 bytes/sec
total size is 10 speedup is 0.07 (DRY RUN)
dst still holds:
one.txt two.txt
✓ the change synced, and the dry-run previewed three.txt without copying it
Applied 1 (your turn) — Disk awareness#
Make two scratch directories of different sizes, compare them with du -sh, and
identify the larger.
496K scratch/heavy
8.0K scratch/light
/dev/root 72G 57G 15G 80% /
largest: scratch/heavy
✓ du sized both directories and identified scratch/heavy as the larger
Applied 2 (your turn) — Symlinks#
Create a symbolic link to a directory, read the -> in ls -l, and follow the link
to reach a file inside the target.
lrwxrwxrwx 1 runner runner 5 Jul 20 06:09 scratch/shortcut -> store
payload
✓ the symlink points at store and resolves through to the real file
Applied 3 (worked) — Background and manage#
Background a long command with &, find it with jobs/ps, then kill it and
confirm it is gone.
[1] 4717
running in background as PID 4717
[1]+ Running sleep 300 &
PID COMMAND
4717 sleep
after kill, still alive? no
[1] 4721
✓ the command backgrounded, appeared in the process table, then was killed
Remote (terminal / conceptual) — ssh and transfer#
No grade here: there is no remote machine on this page. In the Practice here
terminal, generate a key with ssh-keygen and read the connect-and-copy syntax;
then, if you have access to a real cluster (ETH’s Euler, say), do the real
thing:
# [your terminal]
ssh-keygen -t ed25519 # make a key pair
ssh-copy-id you@euler.ethz.ch # install the public half
ssh euler # connect (alias from ~/.ssh/config)
scp run.tar.gz euler:scratch/ # copy a bundle up
rsync -avz results/ euler:results/ # or sync a directory (add -z over the network)
Composite — putting it together (package and sync off the cluster)#
The realistic “get my results off the machine” workflow, composing tar, rsync,
and du. Bundle a results directory, sync the bundle to a destination (a local
directory standing in for the remote), and verify it arrived intact.
sending incremental file list
job-results.tar.gz
sent 344 bytes received 35 bytes 758.00 bytes/sec
total size is 223 speedup is 0.59
4.0K scratch/job-results.tar.gz
--- arrived, contents: ---
output/
output/result.txt
output/run.log
✓ the results were bundled, synced to the destination, and verified intact
Optional stretch (terminal) — tmux, or a careful --delete#
No grade. In the Practice here terminal: start tmux, run something, press
Ctrl-b d to detach, then tmux attach to return. Or practise a mirror sync
(rsync -av --delete --dry-run src/ dst/) and read the would-delete list carefully
before ever dropping the --dry-run (heed the §B admonition).
Outlook#
You can now reach a cluster, carry your scripts and data on and off it, and keep
long work alive and watched after you disconnect: the whole “machine you’re not
at” toolkit. One thing is still missing: on a real cluster you do not just ssh in
and run your job by hand, because hundreds of people share the machine. You hand it
to a scheduler, which queues it and runs it when resources are free. Next
(Notebook 16): writing a SLURM submission script (loading the environment from
Notebook 14 inside it), submitting it with sbatch, and watching the queue.
ssh— open a shell on a remote machine — or run a single command there — over an encrypted connection.scp— copy files to or from a remote machine over ssh — a simple one-shot transfer.rsync— sync files and directories — locally or over ssh — copying only what has changed. The workhorse for moving data on and off a cluster.tar— bundle many files and directories into one archive (and, with -z, compress it) — for transport or backup.gzip— compress a single file in place (and gunzip to restore it) — the compression behind tar's -z.tmux— run a persistent terminal that survives disconnects — detach, log out, and reattach later with your work still running.nohup— run a command immune to hangups, so it keeps going after you log out.ps— take a snapshot of the processes running right now — what is alive, with their PIDs.top— watch processes live — a continuously updating table of what is using the CPU and memory.kill— send a signal to a process — most often to stop one — by its PID.du— report how much disk space files and directories use — for finding what is filling your quota.df— report free and used space on the mounted filesystems — how full the disk (or your cluster quota) is.ln— create a link — most usefully a symbolic link, a lightweight pointer from one path to another.watch— re-run a command every couple of seconds and show its latest output in place — to watch something change.
See the full Compendium Scriptorum for every command met so far, and where to find it again.