1 of 3

Processes, Tuning and LogsLesson 1 of 3

Finding and stopping processes

Read the process list, find what is eating CPU or memory, and stop it with the gentlest signal that works.

  • ps
  • top
  • pgrep
  • pkill
  • kill
  • killall
  • kill -l
  • jobs
  • ps --sort
  • free

40 min5 exercises

After this lesson you can

  • List processes with ps and top and sort them by CPU or memory use.
  • Find a process by name or owner with pgrep, and stop it with kill or pkill using the correct signal.
  • Explain SIGTERM, SIGKILL and SIGHUP, and what states like S, R, D and Z mean.

The idea

A process is a running program with a PID (process ID), an owner, a parent, a state, and resource usage. Administrators spend a good deal of time answering two questions: what is using the machine? and how do I make it stop?

Seeing processes

ToolUse
ps -ef or ps auxA snapshot of every process
ps -eo pid,user,%cpu,%mem,comm --sort=-%cpuYou choose the columns and sort order
topA live, refreshing view; P sort by CPU, M by memory, k kill, q quit
pgrep -a nameFind PIDs by name, with the command line
free -hMemory and swap totals
uptimeLoad averages over 1, 5 and 15 minutes

The load average is the average number of processes wanting CPU or waiting on disk. As a rough rule, a sustained load much higher than the number of CPU cores (nproc) means the machine is overloaded.

The state letter in STAT: R running or ready, S sleeping (waiting for something, normal), D uninterruptible sleep (usually disk or NFS I/O, cannot be killed until the I/O returns), T stopped, Z zombie (already finished, waiting for its parent to collect its exit status).

Signals

Stopping a process is really sending it a signal, and the program decides what to do, except for two.

SignalNumberMeaning
SIGTERM15“Please terminate.” The default of kill. Lets the program clean up
SIGKILL9Immediate, from the kernel. Cannot be caught or ignored; no cleanup
SIGHUP1“Hang up.” Many daemons re-read their configuration on it
SIGSTOP / SIGCONT19 / 18Pause and resume
SIGINT2What Ctrl+C sends

Always try SIGTERM first and give it a few seconds. Use -9 only if the process ignores it, because a killed process cannot remove its lock files, flush buffers or tell its children.

kill 2214            # SIGTERM by PID
kill -9 2214         # SIGKILL
pkill sha256sum      # by name (a pattern matched against the command name)
pkill -u alex        # everything owned by alex
killall httpd        # all processes with this name

Try it

ps -eo pid,user,%cpu,%mem,comm --sort=-%cpu | head -6
    PID USER     %CPU %MEM COMMAND
   2214 alex     97.3  0.0 sha256sum
    842 root      0.2  0.4 sshd
    731 root      0.1  0.6 tuned
sha256sum /dev/zero &
pgrep -a sha256sum
kill $(pgrep sha256sum)
[1] 2214
2214 sha256sum /dev/zero
[1]+  Terminated              sha256sum /dev/zero
top -b -n1 | head -5
top - 11:20:41 up 3:40,  1 user,  load average: 0.95, 0.40, 0.18
Tasks: 128 total,   2 running, 126 sleeping,   0 stopped,   0 zombie
%Cpu(s): 25.0 us,  0.4 sy,  0.0 ni, 74.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   1740.4 total,    812.6 free,    461.2 used,    599.1 buff/cache
MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.   1279.2 avail Mem

The header tells you most of what you need: load average, how many processes exist per state, the CPU split (us user, sy kernel, id idle, wa waiting on I/O) and memory. A high wa points at storage, not CPU. Read avail Mem, not free: Linux uses spare memory for the file cache, which is returned to programs when needed.

Ignoring SIGTERM

bash -c 'trap "" TERM; sleep 600' &
kill %1; sleep 1; jobs
kill -9 %1
[1] 2301
[1]+  Running                 bash -c 'trap "" TERM; sleep 600' &
[1]+  Killed                  bash -c 'trap "" TERM; sleep 600'

By user, by name

pgrep -u alex -a
sudo pkill -u alex sleep

Going further

kill -l lists every signal. renice and nice (next lesson) change priority instead of stopping. systemctl kill --signal=SIGHUP SERVICE signals a unit’s processes, and for a service you should stop it with systemctl stop, not kill, or systemd will restart it. lsof -p PID and ls -l /proc/PID/fd show what a process has open, strace -p PID shows its system calls, and cat /proc/PID/status shows memory details. Out-of-memory kills appear in the journal as Out of memory: Killed process.

Practice

0 of 5 done

Do these on your own system. They get harder. Check each result, then tick it off.

  1. Warm-up

    Who is using the CPU and memory?

    Print the five processes using the most CPU, then the five using the most memory, each with PID, owner, CPU%, memory% and command name.

    Check your result

    Two lists of five lines (plus a header) sorted descending by the chosen column.

    Hint

    ps -eo pid,user,%cpu,%mem,comm --sort=-%cpu | head -6. The minus sign sorts descending.

    Solution
    ps -eo pid,user,%cpu,%mem,comm --sort=-%cpu | head -6
ps -eo pid,user,%cpu,%mem,comm --sort=-%mem | head -6
  2. Practice

    Watch it live

    Start a CPU hog in the background with sha256sum /dev/zero &. In top, find it, sort by memory with M and back to CPU with P, show a single-process view with top -p PID, then quit. Record the PID you will need to kill.

    Check your result

    top shows sha256sum near 100 % CPU. pgrep -a sha256sum prints its PID.

    Hint

    top -b -n1 | head -15 gives a one-shot, non-interactive view. q quits interactive top.

    Solution
    sha256sum /dev/zero &
pgrep -a sha256sum
top -b -n1 | head -12
  3. Practice

    Stop it politely, then forcefully

    Stop the sha256sum process by PID using the default signal, confirm it is gone. Then start another one, stop it by name with a single command, and finally demonstrate when -9 is needed using a process that ignores SIGTERM (create one with bash -c 'trap "" TERM; sleep 600' &).

    Check your result

    After kill PID the process is gone from pgrep. pkill sha256sum does the same by name. kill PID does nothing to the trap-ing shell, and kill -9 PID removes it.

    Hint

    kill PID sends SIGTERM (15). kill -9 sends SIGKILL, which cannot be ignored. Always try 15 first, so the program can clean up.

    Solution
    sha256sum /dev/zero & pid=$!
kill $pid; sleep 1; pgrep sha256sum || echo gone
sha256sum /dev/zero &
pkill sha256sum; sleep 1; pgrep sha256sum || echo gone
bash -c 'trap "" TERM; sleep 600' & pid=$!
kill $pid; sleep 1; ps -p $pid -o pid,comm
kill -9 $pid; sleep 1; ps -p $pid -o pid,comm
  4. Practice

    Kill everything a user is running

    As root, start a sleep 600 as user alex (sudo -u alex sleep 600 &). List alex's processes, then terminate all of them with one command. Explain why pkill -f can be dangerous.

    Check your result

    pgrep -u alex -a sleep shows the process; after sudo pkill -u alex it is gone. pkill -f PATTERN matches the whole command line, so a pattern like sleep can match more than you meant.

    Hint

    pgrep -u USER, pkill -u USER. For a user's login sessions, loginctl terminate-user USER is the cleaner tool.

    Solution
    sudo -u alex sleep 600 &
pgrep -u alex -a
sudo pkill -u alex sleep
pgrep -u alex -a || echo none
  5. Challenge

    Read process states

    List processes with their state letter. Find one of each state you can: running or runnable (R), sleeping (S), and stopped (T, create one with sleep 600 then Ctrl+Z). Explain what a zombie (Z) is and why kill -9 on a zombie does nothing.

    Check your result

    ps -eo pid,stat,comm shows S for most, R for the ps itself, T for the stopped job. A zombie has already exited; only its parent can reap it by reading its exit status.

    Hint

    ps -eo pid,ppid,stat,comm. Ctrl+Z stops the foreground job; bg resumes it in the background, fg returns it.

    Solution
    ps -eo pid,ppid,stat,comm | head
sleep 600      # press Ctrl+Z
jobs
ps -o pid,stat,comm -p $(pgrep -n sleep)
kill %1