3 of 4

Working with TextLesson 3 of 4

Reshaping text: cut, sort, uniq, tr, sed and awk

Select fields, order and count lines, substitute text and compute from columns, all from the command line.

  • cut
  • sort
  • uniq
  • tr
  • paste
  • sed
  • awk

50 min5 exercises

After this lesson you can

  • Extract fields and reorder, de-duplicate and count lines.
  • Replace text in a stream, and edit a file in place safely with sed.
  • Use awk to print, filter and total columns.

The idea

Text tools follow one pattern: read lines, change them, write lines. Learn what each is for and you can combine them without memorising options.

ToolUse it to
cut -d: -f1,7Keep chosen fields (or character columns with -c)
sort, sort -n, sort -r, sort -uOrder lines; numerically, reversed, or unique
uniq, uniq -cCollapse adjacent duplicate lines, optionally counting them
tr 'a-z' 'A-Z', tr -d '\r'Translate or delete single characters
paste, column -tJoin lines side by side, align into columns
sedEdit a stream: substitute, delete, print selected lines
awkTreat each line as fields ($1, $2, $NF) and compute

Try it

cut -d: -f1,3 /etc/passwd | sort -t: -k2 -n | tail -3
alex:1000
bob:1001
nobody:65534
printf '%s\n' pear apple pear fig apple pear | sort | uniq -c | sort -rn
      3 pear
      2 apple
      1 fig

sed: substitute and edit in place

s/old/new/ replaces the first match on each line, and g at the end replaces all of them. Without -i, sed only prints: that is your free preview.

printf 'port=80\nhost=localhost\ndebug=on\n' > ~/lab/app.conf
sed 's/^port=.*/port=8080/' ~/lab/app.conf
sed -i.bak 's/^port=.*/port=8080/' ~/lab/app.conf
diff ~/lab/app.conf.bak ~/lab/app.conf
port=8080
host=localhost
debug=on
1c1
< port=80
---
> port=8080

Other everyday sed commands: sed -n '5,10p' file prints lines 5 to 10, sed '/^#/d' file deletes comment lines, sed '3i\text' inserts a line before line 3.

awk: fields and arithmetic

awk 'pattern { action }' runs the action on lines matching the pattern. Fields are $1, $2 … and $NF is the last. -F: sets the field separator, BEGIN and END run before the first and after the last line.

awk -F: '$3 >= 1000 {print $1, $3}' /etc/passwd
ls -l /etc | awk '/^-/ { sum += $5 } END { print sum " bytes" }'
alex 1000
bob 1001
nobody 65534
312844 bytes

Going further

tr -d '\r' strips the Windows carriage returns that break scripts copied from other systems. sort -t: -k3 -n sorts on field 3 numerically with : as separator. sort -u is sort | uniq. awk '!seen[$0]++' removes duplicates without sorting, and sed -n '/start/,/end/p' prints a range between two patterns.

Practice

0 of 5 done

Do these on your own system. They get harder. Check each result, then tick it off.

  1. Warm-up

    Pull two fields out

    From /etc/passwd, print each user name and its login shell separated by a tab.

    Check your result

    Output lines look like root<TAB>/bin/bash.

    Hint

    cut -d: -f1,7 keeps fields 1 and 7. Add --output-delimiter=$'\t' for a tab, or use awk -F: -v OFS='\t'.

    Solution
    cut -d: -f1,7 --output-delimiter=$'\t' /etc/passwd
  2. Practice

    Order and count

    Create ~/lab/fruit.txt containing pear apple pear fig apple pear, one word per line. Print each distinct fruit with how many times it appears, most frequent first.

    Check your result

    Output is 3 pear, 2 apple, 1 fig, each count right-aligned by uniq -c.

    Hint

    uniq only collapses adjacent duplicates, so sort first. Then sort -rn orders by count.

    Solution
    printf '%s\n' pear apple pear fig apple pear > ~/lab/fruit.txt
sort ~/lab/fruit.txt | uniq -c | sort -rn
  3. Practice

    Replace text, preview first

    Create ~/lab/app.conf with the lines port=80, host=localhost and debug=on. Show on screen what the file would look like with port 8080 and debug off, without changing the file. Then make the change in place and keep a backup named app.conf.bak.

    Check your result

    diff ~/lab/app.conf.bak ~/lab/app.conf shows exactly two changed lines, port and debug.

    Hint

    sed 's/old/new/' prints the result. -i.bak edits in place and saves the original with that suffix. Anchor the pattern with ^.

    Solution
    printf 'port=80\nhost=localhost\ndebug=on\n' > ~/lab/app.conf
sed -e 's/^port=.*/port=8080/' -e 's/^debug=.*/debug=off/' ~/lab/app.conf
sed -i.bak -e 's/^port=.*/port=8080/' -e 's/^debug=.*/debug=off/' ~/lab/app.conf
diff ~/lab/app.conf.bak ~/lab/app.conf
  4. Practice

    Compute from columns

    Using the output of ls -l /etc for regular files only, print the total size in bytes of all files in /etc (not subdirectories). One pipeline with awk.

    Check your result

    A single number. find /etc -maxdepth 1 -type f -printf '%s\n' | paste -sd+ | bc agrees.

    Hint

    In ls -l the size is column 5 and regular files start with -. Add column 5 into a variable and print it in an END block.

    Solution
    ls -l /etc | awk '/^-/ { sum += $5 } END { print sum }'
  5. Challenge

    Build a report from a log

    Make ~/lab/access.log with these lines, then print a table of status codes and how many times each occurs, highest first, using one pipeline.

    10.0.0.5 GET /index.html 200
10.0.0.9 GET /missing 404
10.0.0.5 GET /about.html 200
10.0.0.7 POST /login 500
10.0.0.9 GET /index.html 200
    Check your result

    Output is 3 200, 1 500, 1 404 (the two ties may be in either order).

    Hint

    The status is the last field: awk '{print $NF}'. Then sort, count and sort numerically in reverse.

    Solution
    cat > ~/lab/access.log <<'EOT'
10.0.0.5 GET /index.html 200
10.0.0.9 GET /missing 404
10.0.0.5 GET /about.html 200
10.0.0.7 POST /login 500
10.0.0.9 GET /index.html 200
EOT
awk '{print $NF}' ~/lab/access.log | sort | uniq -c | sort -rn | awk '{print $1, $2}'