3 of 3

Archives and Remote LoginLesson 3 of 3

Securely transferring files: scp, sftp and rsync

Copy files and directories between systems over SSH, and know when to choose each tool.

  • scp
  • sftp
  • rsync
  • scp -r
  • scp -P
  • rsync -av
  • rsync -n

35 min5 exercises

After this lesson you can

  • Copy a file or a whole directory to and from a remote system with scp.
  • Browse and transfer interactively with sftp.
  • Synchronise a directory with rsync, preview with a dry run, and repeat it cheaply.

The idea

All three tools use SSH underneath, so they authenticate and encrypt exactly as ssh does.

ToolBest for
scpOne-off copy of files or a directory
sftpInteractive browsing and transfer, like an FTP client over SSH
rsyncRepeated syncs: it copies only what changed, can preview, and can delete extras

A remote location is written user@host:path. A relative path is relative to the remote user’s home directory.

Try it

echo "hello from node1" > ~/lab/hello.txt
scp ~/lab/hello.txt alex@node2:/tmp/
scp alex@node2:/etc/hostname ~/lab/node2-hostname
hello.txt                                     100%   17     8.1KB/s   00:00
hostname                                      100%    6     3.2KB/s   00:00
scp -r ~/lab/data alex@node2:/tmp/
c.txt                                         100%    6     3.0KB/s   00:00
a.txt                                         100%    4     2.2KB/s   00:00
b.txt                                         100%    4     2.1KB/s   00:00

rsync: preview, copy, repeat

-a (archive) preserves permissions, times and ownership and recurses. -v lists files, -n makes it a dry run. The trailing slash matters: data/ means “the contents of data”, data means “the directory itself”.

rsync -avn ~/lab/data/ alex@node2:/tmp/data-sync/
rsync -av  ~/lab/data/ alex@node2:/tmp/data-sync/
rsync -av  ~/lab/data/ alex@node2:/tmp/data-sync/
sending incremental file list
./
a.txt
b.txt
c.txt

sent 100 bytes  received 25 bytes  250.00 bytes/sec
total size is 15  speedup is 0.12 (DRY RUN)
sending incremental file list
created directory /tmp/data-sync
./
a.txt
b.txt
c.txt

sent 233 bytes  received 90 bytes  646.00 bytes/sec
total size is 15  speedup is 0.05
sending incremental file list

sent 117 bytes  received 12 bytes  258.00 bytes/sec
total size is 15  speedup is 0.12

The third run lists no files because nothing changed. That is why rsync, not scp, is the right tool for backups and deployments.

rm ~/lab/data/c.txt
rsync -avn --delete ~/lab/data/ alex@node2:/tmp/data-sync/
sending incremental file list
deleting c.txt

sent 54 bytes  received 12 bytes  132.00 bytes/sec
total size is 8  speedup is 0.12 (DRY RUN)

sftp

sftp alex@node2
Connected to node2.
sftp> cd /tmp
sftp> put /home/alex/lab/hello.txt uploaded.txt
Uploading /home/alex/lab/hello.txt to /tmp/uploaded.txt
sftp> get /etc/os-release
Fetching /etc/os-release to os-release
sftp> bye

Going further

scp -P 2222 (capital P) sets the port for scp, while ssh uses lowercase -p; for rsync pass -e 'ssh -p 2222'. rsync -z compresses on the wire, --exclude skips files, --progress shows speed and -c compares by checksum instead of time and size. To keep SELinux labels and ACLs on a system backup add -AX. Key-based login, in the SSH Key Authentication skill, removes the password prompts that make scripted transfers awkward.

Practice

0 of 5 done

Do these on your own system. They get harder. Check each result, then tick it off.

  1. Warm-up

    Push a file, pull a file

    Create ~/lab/hello.txt on node1 and copy it to /tmp on node2. Then copy /etc/hostname from node2 to ~/lab/node2-hostname on node1.

    Check your result

    ssh alex@node2 cat /tmp/hello.txt prints the text, and cat ~/lab/node2-hostname prints node2's name.

    Hint

    scp source destination. The remote side is written user@host:path.

    Solution
    echo "hello from node1" > ~/lab/hello.txt
scp ~/lab/hello.txt alex@node2:/tmp/
scp alex@node2:/etc/hostname ~/lab/node2-hostname
  2. Practice

    Copy a directory tree

    Copy the whole directory ~/lab/data (three files) to node2's /tmp. Confirm the files arrived by listing them over SSH, without logging in interactively.

    Check your result

    ssh alex@node2 ls /tmp/data prints the three file names.

    Hint

    Directories need -r.

    Solution
    scp -r ~/lab/data alex@node2:/tmp/
ssh alex@node2 ls /tmp/data
  3. Practice

    Use an interactive sftp session

    Open sftp to node2, change to /tmp, upload ~/lab/hello.txt under the new name uploaded.txt, list the directory, download /etc/os-release to your current local directory, and quit.

    Check your result

    ls /tmp/uploaded.txt exists on node2 and os-release is in your local directory.

    Hint

    Inside sftp use cd, put local remote, ls, get remote, lcd for the local directory and bye.

    Solution
    sftp alex@node2
# cd /tmp
# put /home/alex/lab/hello.txt uploaded.txt
# ls
# get /etc/os-release
# bye
  4. Practice

    Preview, then synchronise

    Synchronise ~/lab/data/ to node2:/tmp/data-sync/. First do a dry run that lists what would be copied, then run it for real, then run it a third time and observe that nothing is copied.

    Check your result

    The dry run prints the file names without creating them. The real run copies them. The third run transfers no files.

    Hint

    rsync -avn is a verbose dry run. A trailing slash on the source copies the contents of the directory.

    Solution
    rsync -avn ~/lab/data/ alex@node2:/tmp/data-sync/
rsync -av  ~/lab/data/ alex@node2:/tmp/data-sync/
rsync -av  ~/lab/data/ alex@node2:/tmp/data-sync/
  5. Challenge

    Mirror exactly, safely

    Make node2:/tmp/data-sync/ an exact mirror of ~/lab/data/: after you delete c.txt locally, a re-sync must also remove it remotely. Preview the deletion before you do it.

    Check your result

    The dry run shows deleting c.txt. After the real run ssh alex@node2 ls /tmp/data-sync no longer lists c.txt.

    Hint

    --delete removes files on the destination that no longer exist at the source. Always add -n first.

    Solution
    rm ~/lab/data/c.txt
rsync -avn --delete ~/lab/data/ alex@node2:/tmp/data-sync/
rsync -av  --delete ~/lab/data/ alex@node2:/tmp/data-sync/
ssh alex@node2 ls /tmp/data-sync