2 of 4

Shell FluencyLesson 2 of 4

Redirection and pipes

Send output to files, keep errors apart from results, and chain small commands into pipelines.

  • >
  • >>
  • 2>
  • &>
  • <
  • |
  • tee
  • xargs
  • /dev/null

40 min5 exercises

After this lesson you can

  • Redirect standard output, standard error and standard input, together or separately.
  • Append instead of overwrite, and avoid destroying a file by accident.
  • Build pipelines with tee, xargs and grep to answer a question in one line.

The idea

Every command has three standard streams: input (0, normally your keyboard), output (1, normally the screen) and errors (2, also the screen). Redirection points a stream somewhere else. A pipe connects the output of one command directly to the input of the next, which is how Linux builds big tools from small ones.

The rule to remember: the shell sets up redirection before it starts the command, so > empties the target file even if the command then fails. That is why sort file > file leaves you with an empty file.

OperatorMeaning
> fileSend output to file, replacing it
>> fileAppend output to file
2> fileSend errors to file
&> fileSend output and errors to file (bash)
2>&1Send errors to wherever output goes
< fileRead input from file
a | bFeed the output of a into b

Try it

cd ~/lab
ls /etc/hostname /nonexistent > ok.txt 2> err.txt
cat ok.txt err.txt
/etc/hostname
ls: cannot access '/nonexistent': No such file or directory
find /etc -name 'ssh*' > found.txt 2> /dev/null
wc -l found.txt
7 found.txt

Order matters with 2>&1: it means “make stream 2 point where stream 1 points right now”, so it goes after the >.

ls /etc/hostname /nonexistent > both.txt 2>&1
cat both.txt
ls: cannot access '/nonexistent': No such file or directory
/etc/hostname

Pipelines

cut -d: -f7 /etc/passwd | sort | uniq -c | sort -rn
     16 /sbin/nologin
      3 /bin/bash
      1 /bin/sync
      1 /sbin/shutdown
      1 /sbin/halt

tee writes its input to a file and passes it on, so you can save the middle of a pipeline. xargs turns lines on its input into arguments of another command.

find /etc -maxdepth 1 -name '*.conf' -readable 2>/dev/null | tee confs.txt | head -3
xargs wc -l < confs.txt | tail -1
/etc/resolv.conf
/etc/sudo.conf
/etc/nsswitch.conf
 1217 total

Going further

A here-document feeds several lines to a command without a temporary file, which is how you write a config file from a script:

cat > ~/lab/motd.txt <<'EOT'
Welcome to rhel10.
Authorised use only.
EOT
cat ~/lab/motd.txt
Welcome to rhel10.
Authorised use only.

Quote the delimiter ('EOT') to stop the shell expanding $variables inside the text. sudo and redirection do not mix: in sudo echo x > /etc/file the redirection is done by your shell, not by root. Use echo x | sudo tee /etc/file instead.

Practice

0 of 5 done

Do these on your own system. They get harder. Check each result, then tick it off.

  1. Warm-up

    Overwrite, then append

    In ~/lab, write the date to log.txt, then add the host name to the same file without losing the date. Show the file.

    Check your result

    cat ~/lab/log.txt prints two lines: a date, then the host name.

    Hint

    > replaces the file. >> adds to the end.

    Solution
    mkdir -p ~/lab && cd ~/lab
date > log.txt
hostname >> log.txt
cat log.txt
  2. Practice

    Separate results from errors

    Run ls /etc/hostname /nonexistent so the successful line goes to ok.txt and the error message goes to err.txt. Neither may appear on screen.

    Check your result
    cat ok.txt err.txt

    ok.txt holds /etc/hostname. err.txt holds ls: cannot access '/nonexistent': No such file or directory.

    Hint

    Standard output is file descriptor 1, standard error is 2. 2> redirects only stderr.

    Solution
    ls /etc/hostname /nonexistent > ok.txt 2> err.txt
cat ok.txt err.txt
  3. Practice

    Keep the results, silence the errors

    Run find /etc -name 'ssh*' as a normal user. Send the results to found.txt, and throw away the "Permission denied" errors so they neither show on screen nor land in the file. Count the lines in found.txt.

    Check your result

    wc -l found.txt prints a number greater than zero and no error appears on screen. grep -c denied found.txt prints 0.

    Hint

    /dev/null discards everything written to it. Redirect only stream 2 there.

    Solution
    find /etc -name 'ssh*' > found.txt 2> /dev/null
wc -l found.txt
grep -c denied found.txt
  4. Practice

    Top three shells in use

    From /etc/passwd, print the three most common login shells, most common first, each with its count. One pipeline.

    Check your result

    Output is three lines, each a count followed by a shell path, with the biggest count first, for example 16 /sbin/nologin.

    Hint

    cut -d: -f7 takes the shell field, sort then uniq -c counts, sort -rn orders, head -3 trims.

    Solution
    cut -d: -f7 /etc/passwd | sort | uniq -c | sort -rn | head -3
  5. Challenge

    Save and show; run a command per result

    List the files directly in /etc whose names end in .conf and that you can read. Show the first five on screen and save the full list to confs.txt in the same command. Then, using the saved list, print the number of lines of every one of those files, largest last.

    Check your result

    confs.txt has the full list. The second command prints a line count for each file and ends with a total line, with the biggest file just before it.

    Hint

    tee copies its input to a file and to the screen. xargs wc -l turns lines of file names into arguments.

    Solution
    find /etc -maxdepth 1 -name '*.conf' -readable 2>/dev/null | tee confs.txt | head -5
xargs wc -l < confs.txt | sort -n