File Transfer — scp and rsync

Learning objectives

• Understand why and when you need to move data between your computer and the cluster
• Copy files and directories between your local computer and the lab417 server using scp
• Use rsync for smarter, resumable transfers that copy only what has changed
• Choose the right tool for single files versus directories and repeated transfers

Approximate time: 20 minutes

Why do we need to transfer data?

A typical bioinformatics workflow is split across two machines:

• Your local computer (your laptop) — where you read papers, view results, and make figures.
• The lab417 server — where the heavy computing happens: alignment, assembly, variant calling.

So you constantly need to move data in both directions: up to the server (raw sequencing reads, reference genomes, scripts) and down to your laptop (assemblies, QC reports, plots). Both scp and rsync do this securely over the same SSH connection you already use to log in.

Important: all of these commands are run from your local computer's terminal, not from inside an SSH session on the server. If your prompt shows username@lab417, you are on the server — open a new local terminal first.

Windows users — MobaXterm

Good news: MobaXterm bundles both scp and rsync built-in, so Windows participants run the exact same commands as Mac and Linux users — no extra installation needed. When MobaXterm opens an SSH session it also spins up a local Unix-like terminal on your Windows machine, where these commands work just as shown below.

Where do downloaded files land?

When you copy a file down to "here" (./), it lands in MobaXterm's local home folder, which maps to something like:

C:\Users\username\Documents\MobaXterm\home\

Set this before the course. Open MobaXterm Settings → General and set a sensible local working directory (your "persistent home directory" / download path) ahead of time, so your downloaded files always land somewhere you can find them — not lost in a temporary folder.

Drag-and-drop alternative

MobaXterm also shows a file browser panel on the left that displays the server's files graphically. For a single file you can simply drag it between this panel and your Windows folders. Many beginners find this easier for their first few transfers — but learn the commands too, as they are what you will script and use on every other system.

Note on rsync in MobaXterm: it is included, but depending on the MobaXterm version it can occasionally be flaky. Test it before the course. If it does not work on your machine, scp -r covers the same directory-copy use case for the practical.

Copying files with scp

The scp (secure copy) command copies files over SSH. It looks just like the cp command you already know, except one side is a remote location written as:

username@host:path

The general form is:

scp  SOURCE  DESTINATION

Copying a file up to the server

Send a local FASTQ file to your data directory on the server:

scp localfile.fastq username@lab417.saiab.ac.za:~/data/

Here the source is a plain local file and the destination is remote. The ~ is your home directory on the server.

Copying a file down from the server

Pull an assembly back to your current local directory. The single dot (.) means "right here":

scp username@lab417.saiab.ac.za:~/results/assembly.fasta ./

Copying a whole directory

To copy a directory and everything inside it, add the -r (recursive) flag — exactly like cp -r:

scp -r username@lab417.saiab.ac.za:~/results/ ./results/

Tip: you will be prompted for your lab417 password each time, just like when you log in. Watch the direction carefully — the side with the username@host: prefix is always the server.

Smarter transfers with rsync

scp always copies everything, every time. If a transfer is interrupted, you start again from scratch. rsync is smarter: it compares source and destination and only transfers the parts that are new or changed. This makes it ideal for large datasets and for transfers you repeat as results accumulate.

A typical command to sync a results directory down from the server:

rsync -avh username@lab417.saiab.ac.za:~/results/ ./results/

Common rsync flags

• -a — archive mode: recurse into directories and preserve permissions, timestamps, and symbolic links
• -v — verbose: list each file as it is transferred
• -h — human-readable file sizes (KB, MB, GB instead of raw bytes)
• --progress — show a live progress bar for each file
• --dry-run — preview what would be copied without actually copying anything

Putting the useful ones together:

rsync -avh --progress username@lab417.saiab.ac.za:~/results/ ./results/

Always dry-run first. Before a big or unfamiliar transfer, add --dry-run to see exactly what rsync intends to do. When the output looks right, remove the flag and run it for real.

Watch the trailing slash. A trailing / on the source directory (~/results/) copies the contents of results into the destination. Without the slash (~/results) it copies the results folder itself into the destination. When in doubt, --dry-run.

When to use scp vs. rsync

• Use scp for a quick, one-off copy of a single file.
• Use rsync for directories, large datasets, and any transfer you might repeat — it resumes and skips unchanged files.

Exercises

1. Transfer a FASTQ file from the server to your local machine using scp.
2. Now transfer the same file with rsync -avh --progress. Run it a second time immediately — what does rsync report, and why is it almost instant?
3. Use rsync --dry-run -avh on a directory and read the output. What would have been copied?
4. Copy a whole results directory from the server to your laptop using scp -r.

SAIAB AGRP Bioinformatics Training. Open access materials distributed under the terms of the Creative Commons Attribution license (CC BY 4.0).