Monitoring a system's vital signs

The idea

Monitoring answers “is it healthy, and if not, which resource is short?”. A good habit is the USE method: for each resource (CPU, memory, disk, network) ask three questions in order.

	Question	Example sign
Utilisation	How busy is it?	CPU 95 % busy, disk 100 % %util
Saturation	Is work waiting for it?	Run queue longer than the CPU count, swap in/out, long I/O wait
Errors	Is it failing?	Disk or NIC errors, OOM kills, dmesg lines

High utilisation on its own is not a fault; saturation and errors are. A CPU at 100 % that finishes all its work is fine. A CPU at 60 % with a long run queue is not.

CPU and load

uptime prints three load averages (1, 5 and 15 minutes): the average number of threads that were running, waiting to run, or (on Linux) waiting uninterruptibly for I/O. Compare it with the number of CPUs from nproc. On 2 CPUs a load of 1.5 is fine and a load of 6 means work is queueing. Three numbers show the direction: 0.2, 2.1, 5.0 is a problem that is getting better, 5.0, 2.1, 0.2 one that is starting.

vmstat 1 prints one line per second. The columns you need:

Column	Meaning	Worry when
r	Runnable threads (the run queue)	Greater than the number of CPUs for long
b	Threads blocked in uninterruptible sleep (usually I/O)	Persistently above 0
si, so	KiB swapped in and out per second	Continuously non-zero
bi, bo	Blocks read and written	Compare with disk limits
us, sy	CPU in user and kernel code	us+sy near 100 with r high: CPU bound
id	Idle	Near 0 while r is high
wa	Waiting for I/O	High: disk or network file system is slow
st	Stolen by the hypervisor	High on a VM: the host is overcommitted

top (press 1 for every CPU, M to sort by memory, P by CPU) and ps -eo pid,%cpu,%mem,rss,comm --sort=-%cpu tell you who. vmstat tells you what kind of problem; use both.

Since Linux 4.20 the kernel also reports pressure stall information: /proc/pressure/cpu, memory and io say how much time tasks spent stalled waiting for that resource. some avg10=12.00 means that over the last 10 seconds, 12 % of the time at least one task was stalled. It is a direct saturation measurement and it exists even when the load average is ambiguous.

Memory

               total        used        free      shared  buff/cache   available
Mem:           7.8Gi       479Mi       7.3Gi        15Mi       295Mi       7.4Gi

free is memory that nothing uses. Linux uses spare memory as file cache, so a healthy long-running system shows little free. available estimates what applications could get without swapping, because the cache can be dropped. Low available, rising swap activity (si/so) and OOM-killer lines in dmesg are the real warnings.

journalctl -k -b -g "Out of memory|oom-kill" --no-pager

Disk space, inodes and deleted files

A file system can be “full” in three ways, and each needs a different command.

Symptom	Check	Cause
No space left on device	df -hT	Blocks used up
Same message but df -h shows free space	df -i	Inodes used up (millions of tiny files)
df says full but du cannot find the files	lsof +L1	Deleted files still held open by a process
Directory is large, which one?	du -xh --max-depth=1 /var | sort -h | tail	-x stays on one file system

The third case catches many administrators. rm removes a name; the space is released only when no process has the file open. A service that keeps writing to a log you deleted keeps the space. lsof +L1 lists open files with a link count below 1.

Disk I/O and network

iostat -xz 1 (from the sysstat package) shows per-device r/s, w/s, await (average milliseconds per request) and %util. A device at %util near 100 with a growing await is saturated. sar -n DEV 1 and ip -s link show packets, errors and drops per interface; ss -s summarises sockets.

Keeping history with sysstat

Most real problems are over by the time you look. The sysstat package records sar data every 10 minutes (sysstat-collect.timer) and keeps it for 28 days under /var/log/sa/.

sudo dnf install -y sysstat
sudo systemctl enable --now sysstat
sar -u -s 09:00:00 -e 10:00:00     # CPU between 09:00 and 10:00 today
sar -r                             # memory today
sar -q                             # run queue and load today

Try it

This is real output from a two-CPU machine while two yes processes ran.

vmstat 1 3
top -b -n1 | head -12

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 3  0      0 7634716  15320 288140    0    0   979    60  119    0  2  1 97  0  0  0
 2  0      0 7634456  15320 288248    0    0   168     0  541  104 73 24  0  0  3  0
 2  0      0 7644792  15320 288316    0    0     0     0  519   69 75 25  0  0  1  0
top - 16:49:33 up 5 min,  0 user,  load average: 0.19, 0.08, 0.02
Tasks:  67 total,   3 running,  64 sleeping,   0 stopped,   0 zombie
%Cpu(s): 66.7 us, 33.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  268 root      20   0    2728   1580   1452 R 100.0   0.0   0:04.12 yes
  266 root      20   0    2728   1576   1452 R  90.9   0.0   0:04.12 yes

Read it like this. The first vmstat line is the average since boot; ignore it. The later lines show r of 2 on a 2-CPU machine, us+sy at 98 and id 0: the CPUs are fully utilised. Saturation is only starting (r equals the CPU count). The load average is still 0.19 because it moves slowly: that is why vmstat and PSI are better than load for “what is happening right now”.

cat /proc/pressure/cpu

some avg10=0.77 avg60=0.15 avg300=0.07 total=910939
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

The deleted-but-open file

df -h /var/tmp; du -sh /var/tmp
sudo lsof +L1 | head -3

/dev/vda        252G   13G   30G  31% /var/tmp
8.0K	/var/tmp
COMMAND PID USER   FD   TYPE DEVICE   SIZE/OFF NLINK    NODE NAME
bash    287 root    4w   REG  254,0   52428800     0 1204595 /var/tmp/big.log (deleted)

NLINK 0 and (deleted) are the signature. The data is still on disk and counted by df, but invisible to du because no directory entry points to it. Free it by restarting the service that holds it, or truncate through the descriptor with : > /proc/PID/fd/N. Then fix the cause: log rotation that deletes instead of using copytruncate or a reload signal.

When it goes wrong

Observation	Probable cause	Next check
Load 12 on 2 CPUs, CPU idle 60 %, b high	Threads blocked on I/O	iostat -xz 1, wa, /proc/pressure/io
Everything slow, si/so non-zero, available low	Memory shortage	ps --sort=-rss, dmesg for OOM
st above 10 on a VM	Host overcommitted	Nothing to fix inside the guest; tell the host owner
df and du disagree by a lot	Deleted open files, or data under a mount point	lsof +L1, umount the mount and look underneath
Disk %util 100, low throughput	Failing device or a slow remote file system	dmesg, smartctl -a if available

Going further

Beyond the exam. pidstat 1 (per-process CPU, memory and I/O), iotop, perf top and Performance Co-Pilot (pmstat, pmrep) go deeper. They are not required, but pidstat -d 1 is the fastest way to find which process is writing to a busy disk, and PCP is what the web console’s history graphs use (next lesson).