3 of 4

Shell ScriptingLesson 3 of 4

Repeating work: for, while and until

Do the same thing to a list of names, files or lines, and stop or skip when you need to.

  • for
  • while
  • until
  • read
  • break
  • continue
  • seq
  • {1..5}
  • ${var%.txt}

40 min5 exercises

After this lesson you can

  • Loop over a list, a glob, a range and the arguments of a script with for.
  • Read a file line by line, splitting fields, with while read.
  • Control a loop with break and continue, and wait for a condition with until.

The idea

Administration is repetition: the same change on ten users, the same check on every file in a directory, the same step until something is ready. Bash has three loops.

LoopRunsUse it for
for x in LISTOnce per item in the listNames, files, arguments, a range
while CONDITIONWhile the condition succeedsReading lines, counters
until CONDITIONUntil the condition succeedsWaiting for something to become ready

Inside any loop, break leaves it immediately and continue skips to the next round.

What a for loop can iterate over

for u in alice bob carol; do ...; done       # a literal list
for i in {1..5}; do ...; done                # a range
for f in /etc/*.conf; do ...; done           # a glob, expanded by the shell
for a in "$@"; do ...; done                  # the script's arguments
for u in $(cut -d: -f1 /etc/passwd); do ...  # words of command output

The last form splits output on whitespace, which is fine for user names and breaks on file names with spaces. For line-oriented data prefer while read.

Reading a file line by line

read takes one line from standard input and splits it into variables. Feed the file in at the end of the loop:

while IFS=: read -r name uid shell; do
    echo "$name -> $shell"
done < users.txt

IFS=: makes the colon the separator (for this one read), -r stops backslashes being interpreted, and the last variable receives the rest of the line. This is exactly how you walk /etc/passwd, which is colon-separated.

Try it

for u in alice bob carol; do echo "user: $u"; done
for i in {1..3}; do echo -n "$i "; done; echo
user: alice
user: bob
user: carol
1 2 3 
mkdir -p ~/lab/ren && cd ~/lab/ren
touch a.txt b.txt c.txt
for f in *.txt; do mv "$f" "${f%.txt}.bak"; done
ls
a.bak  b.bak  c.bak

${f%.txt} removes the shortest match of .txt from the end of $f. ${f#prefix} does the same at the start. They are built into Bash, so no sed or basename process is started for each file.

A dry run before a destructive loop

Whenever a loop will change or delete things, write it first so it only prints what it would do. Run that, read it, then swap the echo for the real command.

cat > adduse.sh <<'EOT'
#!/bin/bash
for u in "$@"; do
    if id "$u" &>/dev/null; then
        echo "skip $u (exists)"
    else
        echo "would create $u"
    fi
done
EOT
chmod +x adduse.sh
./adduse.sh root zed
skip root (exists)
would create zed

Splitting fields while reading

printf 'alex:1001:/bin/bash\nbob:1002:/sbin/nologin\n' > users.txt
while IFS=: read -r name uid shell; do
    echo "$name -> $shell"
done < users.txt
alex -> /bin/bash
bob -> /sbin/nologin

Counters, skipping and stopping

for i in 1 2 3 4 5; do
    [ $i -eq 2 ] && continue
    [ $i -eq 4 ] && break
    echo $i
done
1
3

$(( ... )) does integer arithmetic, so a counter is n=$((n+1)), and until is the mirror of while: until [ -f /tmp/ready ]; do sleep 1; done waits for a file to appear. Always put a limit on a waiting loop, or a missing file hangs the script forever.

Going further

for ((i=0; i<5; i++)); do ...; done is the C-style form. Arrays hold lists safely: arr=(one "two words" three), "${arr[@]}" expands each element as its own word, and ${#arr[@]} is the count. select builds a numbered menu. For a command applied to many files, find ... -exec or xargs are often shorter than a loop and handle huge lists better, but a loop is easier to read, debug and extend, which is what matters in an exam script.

Practice

0 of 5 done

Do these on your own system. They get harder. Check each result, then tick it off.

  1. Warm-up

    Loop over a list and a range

    In one script print user: alice, user: bob, user: carol using a for over a list, then print the numbers 1 to 5 on one line using a brace range.

    Check your result

    Three user: lines, then 1 2 3 4 5.

    Hint

    for u in alice bob carol; do ... done, and for i in {1..5}. echo -n suppresses the newline.

    Solution
    for u in alice bob carol; do echo "user: $u"; done
for i in {1..5}; do echo -n "$i "; done; echo
  2. Practice

    Rename a batch of files

    In an empty directory create a.txt, b.txt, c.txt. With a loop rename every .txt file to .bak, keeping the base name. Do not use any external command except mv.

    Check your result

    ls shows a.bak b.bak c.bak and no .txt files.

    Hint

    ${f%.txt} strips the suffix from $f. Quote "$f" in case a name has a space.

    Solution
    mkdir -p ~/lab/ren && cd ~/lab/ren
touch a.txt b.txt c.txt
for f in *.txt; do mv "$f" "${f%.txt}.bak"; done
ls
  3. Practice

    Check several accounts in one run

    Write ~/lab/adduse.sh that takes any number of user names. For each one it prints skip NAME (exists) if the account exists, otherwise would create NAME. It must not create anything: this is a dry run. Run it with root and a name that does not exist.

    Check your result

    ./adduse.sh root zed prints skip root (exists) and would create zed.

    Hint

    for u in "$@"; do if id "$u" &>/dev/null; then ... fi; done.

    Solution
    cat > ~/lab/adduse.sh <<'EOT'
#!/bin/bash
for u in "$@"; do
    if id "$u" &>/dev/null; then
        echo "skip $u (exists)"
    else
        echo "would create $u"
    fi
done
EOT
chmod +x ~/lab/adduse.sh
~/lab/adduse.sh root zed
  4. Practice

    Read a file line by line

    Create ~/lab/users.txt with the lines alex:1001:/bin/bash and bob:1002:/sbin/nologin. With a while read loop print NAME -> SHELL for each line, splitting on the colon.

    Check your result

    Output is alex -> /bin/bash and bob -> /sbin/nologin.

    Hint

    while IFS=: read -r name uid shell; do ... done < users.txt. IFS=: sets the field separator for that read only.

    Solution
    printf 'alex:1001:/bin/bash\nbob:1002:/sbin/nologin\n' > ~/lab/users.txt
while IFS=: read -r name uid shell; do
    echo "$name -> $shell"
done < ~/lab/users.txt
  5. Challenge

    Wait for something, but not forever

    Write ~/lab/wait-for.sh FILE that checks once a second whether FILE exists, for at most 10 tries. It prints found FILE and exits 0 as soon as it exists, or gave up on FILE and exits 1 after 10 failed tries. Test both outcomes, creating the file from a second terminal (or with ( sleep 3; touch ... ) &) in the first.

    Check your result

    With the file created after 3 seconds the script prints found and echo $? shows 0. With no file it prints gave up after about 10 seconds and the status is 1.

    Hint

    Use a counter with $((n+1)), sleep 1, and break or an early exit 0 when found.

    Solution
    cat > ~/lab/wait-for.sh <<'EOT'
#!/bin/bash
file="$1"
for try in $(seq 1 10); do
    if [ -e "$file" ]; then
        echo "found $file"
        exit 0
    fi
    sleep 1
done
echo "gave up on $file" >&2
exit 1
EOT
chmod +x ~/lab/wait-for.sh
( sleep 3; touch /tmp/ready ) &
~/lab/wait-for.sh /tmp/ready; echo $?
rm -f /tmp/ready
~/lab/wait-for.sh /tmp/ready; echo $?