Disk usage: where the space went

Free space: df

df (disk free) reports on filesystems. It asks each one how many blocks it has and how many are used:

df -h

Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.6G  1.9M  1.6G   1% /run
/dev/sda2       234G   87G  135G  40% /
tmpfs           7.8G     0  7.8G   0% /dev/shm
/dev/sda1       511M  6.1M  505M   2% /boot/efi

-h prints sizes in powers of 1024. The columns:

Column	Meaning
Filesystem	The device or source
Size	Total capacity
Used / Avail	Allocated and free blocks
Use%	Used divided by used plus available
Mounted on	Where in the tree it is attached

Ask about a path and df reports the filesystem holding it, which answers “which disk is this on?”:

df -hT ~

Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda2      ext4  234G   87G  135G  40% /

Two useful filters: -x tmpfs hides memory-backed filesystems, and --output= chooses columns:

df --output=source,fstype,size,used,avail,pcent,target /

Filesystem     Type 1K-blocks     Used    Avail Use% Mounted on
/dev/sda2      ext4 245112832 91275264 141418496  40% /

Usage by directory: du

df cannot tell you what is using the space. du (disk usage) walks a directory and adds up the files. Create a sample tree to practise on:

mkdir -p ~/lab/du-lab/{logs,cache,media}
cd ~/lab/du-lab
head -c 3M    /dev/zero > logs/app.log
head -c 200K  /dev/zero > logs/old.log
head -c 12M   /dev/zero > media/clip.bin
head -c 1500K /dev/zero > media/photo.bin
head -c 40K   /dev/zero > cache/c1
head -c 8K    /dev/zero > cache/c2

A total for one directory:

du -sh .

17M	.

-s summarises, -h is human-readable. To see where it goes, go one level down and sort. sort -h understands K, M and G, and -r puts the largest first:

du -h --max-depth=1 . | sort -rh

17M	.
14M	./media
3.2M	./logs
52K	./cache

Repeat that one level at a time, always into the biggest directory, and you find the cause of almost any full disk. This is faster than reading a long listing.

To rank files, list everything with -a:

du -ah . | sort -rh | head -6

17M	.
14M	./media
12M	./media/clip.bin
3.2M	./logs
3.0M	./logs/app.log
1.5M	./media/photo.bin

Option	Effect
-s	One total per argument
-h	Human-readable
-d N / --max-depth=N	Stop N levels down
-x	Stay on one filesystem
-a	Include files, not only directories
--exclude='*.log'	Skip matching names
-c	Add a grand total

-x matters when you start at /: without it du also walks into other disks and into /proc.

Find big files

When you want files, not directories, find selects by size:

find ~/lab/du-lab -type f -size +1M -exec ls -lh {} +

-rw-rw-r-- 1 alex alex 3.0M Nov  3 10:30 /home/alex/lab/du-lab/logs/app.log
-rw-rw-r-- 1 alex alex  12M Nov  3 10:30 /home/alex/lab/du-lab/media/clip.bin
-rw-rw-r-- 1 alex alex 1.5M Nov  3 10:30 /home/alex/lab/du-lab/media/photo.bin

+1M means more than one mebibyte, -1M less than. Units are k, M and G. Combine with -mtime +30 (older than 30 days) or -name '*.log' to narrow it down. -exec CMD {} + runs the command on many results at once.

A single line gives you the top consumers, ready to act on:

find ~/lab/du-lab -type f -exec du -h {} + | sort -rh | head -3

12M	/home/alex/lab/du-lab/media/clip.bin
3.0M	/home/alex/lab/du-lab/logs/app.log
1.5M	/home/alex/lab/du-lab/media/photo.bin

Why the numbers disagree

Three tools, three questions.

Tool	Reports
ls -l	The length of a file, in bytes
du	The blocks a file occupies
df	The blocks the filesystem has allocated

Length and blocks differ because storage comes in blocks. A 6-byte file takes one 4 KiB block. A sparse file has holes that are never allocated:

truncate -s 1G sparse.img
ls -lh sparse.img
du -h sparse.img

-rw-rw-r-- 1 alex alex 1.0G Nov  3 10:30 sparse.img
0	sparse.img

du --apparent-size makes du report lengths instead of blocks.

df says full, du says it isn’t

The most common mismatch: df reports far more used than du can find. The usual causes:

1. A deleted file is still open. Removing a name does not free the data while a process holds the file open. du cannot see it (it has no name) but df counts it.
2. Files hidden under a mount point. Data written to /data before a disk was mounted on it is still there, invisible, under the mount.
3. Reserved blocks and metadata. The journal and reserved space count in df, not in du.

Case 1 is easy to reproduce:

cd ~/lab
seq -f "line %g" 1 200000 > big.txt
tail -f big.txt > /dev/null &
pid=$!
rm big.txt
ls -l /proc/$pid/fd | grep deleted

lr-x------ 1 alex alex 64 Nov  3 10:41 3 -> /home/alex/lab/big.txt (deleted)

Every open file of a process appears in /proc/PID/fd. Here the target is marked (deleted). System-wide, lsof +L1 lists every open file with no remaining name. The space comes back when the process closes the file:

kill $pid

For a long-running service the fix is to restart it, or to truncate the file before deleting it (: > file), which frees the data immediately.

Full without being full: inodes

A filesystem has a fixed number of inodes, chosen when it was created. Each file, directory and link uses one. With millions of tiny files you can run out of inodes with plenty of blocks left.

df -i /

Filesystem       Inodes  IUsed    IFree IUse% Mounted on
/dev/vda       16777216 344026 16433190    3% /

The signature is an error that contradicts df -h. Here a tiny filesystem was made with only 12 inodes (lesson 6 shows how to mount one), and creating the thirteenth file fails:

touch f13

touch: cannot touch 'f13': No space left on device

df -h /mnt/small
df -i /mnt/small

Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.0M     0  1.0M   0% /mnt/small
Filesystem     Inodes IUsed IFree IUse% Mounted on
tmpfs              12    12     0  100% /mnt/small

Zero percent of the bytes used, one hundred percent of the inodes. The fix is to delete files, and the question is where the millions of small files live. Mail queues, session directories and caches are the usual suspects. Count entries per directory to find them:

for d in /var/*; do echo "$(sudo find "$d" -xdev | wc -l) $d"; done | sort -rn | head -3

A habit for full disks

1. df -h to see which filesystem is full.
2. df -i to rule out inodes.
3. du -xh --max-depth=1 on the mount point, then go into the biggest directory, and repeat.
4. If du finds less than df reports, look for deleted-but-open files and for data under mount points.
5. Delete or compress, and check df again.