LUKS headers, key slots and a file system that is 'full'

The idea

Two storage problems look nothing alike but share a lesson: the failure is in a small piece of metadata, not in the data. A LUKS volume is unreadable if its header is damaged, however intact the encrypted blocks are. A file system can report “no space” while most of its bytes are free, because space is not one resource.

LUKS in one page

LUKS (Linux Unified Key Setup) stores encryption metadata in a header at the start of the device. The header holds the cipher settings, a random master key encrypted separately by each of up to 32 (LUKS2) key slots, one per passphrase or key file, and a UUID. The data area follows the header. To open the volume, cryptsetup derives a key from your passphrase, uses it to decrypt a slot, gets the master key, and maps the data area as /dev/mapper/NAME.

Consequences you must remember:

• The header is a single point of failure. If it is overwritten, nothing, not even the right passphrase, can recover the master key. A header backup is the only way back.
• Passphrases are independent slots. Adding a passphrase adds a slot; removing one deletes only that slot. This is how you rotate passwords without re-encrypting.
• Removing the last slot makes the data permanently unreadable. cryptsetup refuses without --force-style confirmation in many cases; do not rely on that.

Command	Purpose
cryptsetup luksDump DEV	Show header: version, UUID, cipher, key slots
cryptsetup isLuks DEV	Exit status 0 if a LUKS header exists (script-friendly)
cryptsetup luksHeaderBackup DEV --header-backup-file F	Save the header
cryptsetup luksHeaderRestore DEV --header-backup-file F	Write it back
cryptsetup luksAddKey DEV [keyfile]	Add a passphrase or key file
cryptsetup luksRemoveKey DEV	Remove the passphrase you type
cryptsetup open DEV NAME / close NAME	Map and unmap (luksOpen/luksClose are the old names)
cryptsetup open --test-passphrase DEV	Verify a passphrase without opening

Opening at boot is controlled by /etc/crypttab, with one line per volume: NAME DEVICE(by UUID) keyfile-or-none options. The mount of the result is a normal /etc/fstab line using /dev/mapper/NAME. If you change either, systemctl daemon-reload and, for the root file system, rebuild the initramfs (dracut -f); see the earlier boot lessons.

Try it

All outputs here are real, from cryptsetup on an image file (a deliberately low iteration count was used to make the test quick; production volumes use the defaults).

The header

cryptsetup luksDump c.img | head -14

LUKS header information
Version:       	2
Epoch:         	3
Metadata area: 	16384 [bytes]
Keyslots area: 	16744448 [bytes]
UUID:          	d8bfff35-9d5c-4876-bd9b-78ded3c47389
Label:         	(no label)
Subsystem:     	(no subsystem)
Flags:       	(no flags)

Data segments:
  0: crypt
	offset: 16777216 [bytes]
	length: (whole device)

The data offset (16 MiB) is the size of the header area. That is also the size of the header backup file:

cryptsetup luksHeaderBackup c.img --header-backup-file hdr.bak
ls -l hdr.bak

-r-------- 1 root root 16777216 Nov  4 17:11 hdr.bak

Destroy and restore the header

dd if=/dev/zero of=c.img bs=1M count=1 conv=notrunc
cryptsetup luksDump c.img | head -3
cryptsetup isLuks c.img; echo isLuks=$?
cryptsetup luksHeaderRestore c.img --header-backup-file hdr.bak -q
cryptsetup isLuks c.img; echo isLuks=$?

Device c.img is not a valid LUKS device.
isLuks=1
isLuks=0

(The dd summary line is omitted; the first command printed no luksDump header because the device is no longer LUKS.) One mebibyte of zeros erased the header and the volume was unreadable until the backup went back. In production, store the backup off the machine, protected as strictly as the passphrase.

Key slots

printf 'labpass\nsecond-pass\n' | cryptsetup luksAddKey c.img
cryptsetup luksDump c.img | grep -E '^\s+[0-9]+: luks2'

  0: luks2
  1: luks2

Two slots now. Test the new passphrase with cryptsetup open --test-passphrase, and only then luksRemoveKey the old one. The order matters: remove first and you may be left with a passphrase you mistyped.

A file system that is “full”

No space left on device has several causes. Check them in this order.

Check	Command	Meaning when exhausted
Blocks	df -h /mnt/t	Data blocks used up (including reserved blocks for root on ext4)
Inodes	df -i /mnt/t	Too many files; blocks may be free
Deleted but open files	ls -l /proc/PID/fd, lsof +L1	Space is held until the last process closes the file
Quotas	repquota -a, xfs_quota -x -c report	Per-user or per-project limit, the file system itself is not full
Thin pools	lvs -o +data_percent,metadata_percent	A thin LV looks free but the pool is full

Inodes

Real output: an ext4 file system created with only 208 inodes, after a loop of empty files:

touch m/x
df -h m | tail -1
df -i m

touch: cannot touch 'm/x': No space left on device
/dev/loop0       60M   24K   56M   1% /tmp/fsr/m
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop0        208   208     0  100% /tmp/fsr/m

The error message is the same as for full blocks, while df -h shows 1 % used. Only df -i shows the cause. Remove the many small files (find the directory: for d in /var/*; do echo "$(find "$d" | wc -l) $d"; done | sort -n | tail). On XFS inodes are allocated dynamically, so this is mostly an ext4 problem, but a fixed-size file system created for millions of small files with the wrong -i option still hits it.

Deleted but still open

A 40 MiB file was written, a process opened it, then the file was deleted:

df -h m | tail -1
ls -l /proc/$P/fd | grep deleted
kill $P
df -h m | tail -1

/dev/loop0       60M   41M   16M  73% /tmp/fsr/m
lr-x------ 1 root root 64 Nov  4 17:11 0 -> /tmp/fsr/m/big (deleted)
/dev/loop0       60M   24K   56M   1% /tmp/fsr/m

du found nothing in the directory, but df still counted 41 MiB, because the process kept the file open. Ending the process (or restarting the service that owns it) released the space immediately. This is a typical result of deleting a huge log file while the daemon still writes to it. Better: truncate in place (: > /var/log/big.log) or use logrotate, which signals the service to reopen its file.

Going further

Why XFS stays quiet about inodes. XFS creates inodes as needed, bounded by imaxpct (a percentage of space). Beyond the exam: LUKS also supports a detached header on separate media, token-based unlocking (TPM2, network-bound with Clevis and Tang), and systemd-cryptenroll. A header backup protects against accidental damage; it does not replace backups of the data itself.