Repairing XFS and ext4 file systems

The idea

A file system is a set of on-disk structures that must agree with each other: a superblock (the description of the whole file system), allocation maps (which blocks are free), inodes (the files) and a journal or log (a record of changes in progress, so a crash does not leave half-finished updates). Repair tools check that these agree and fix what does not.

Three rules decide the order of work.

1. Understand the failure before repairing. A mount that fails because the device is missing, the wrong type is named, or the LVM volume is inactive is not a corrupted file system, and repair tools will not help. Check the layers below the file system first.
2. Never repair a mounted file system. The tools modify structures the kernel is using. Unmount, or boot rescue media.
3. Look before you change. Run the read-only check (xfs_repair -n, e2fsck -fn) first. Repair changes data; make a copy of anything irreplaceable if you can.

What the failure tells you

lsblk -f               # does the kernel see the device and a file system signature?
sudo blkid DEVICE      # what does the signature say? Nothing printed = no recognised file system
mount DEVICE /mnt      # the error
dmesg | tail           # the kernel's reason

mount says	blkid says	Meaning	Next
special device ... does not exist		Device is missing, or LVM LV is not active	lsblk, lvs, vgchange -ay
wrong fs type, bad option, bad superblock	nothing	Superblock unreadable	Superblock recovery (below)
wrong fs type...	a type different from the one you used	You named the wrong type	mount -t TYPE or omit it
Structure needs cleaning	xfs	XFS metadata corruption detected	xfs_repair
mount: ... can't read superblock	ext4	ext4 superblock/group descriptors damaged	e2fsck, backup superblock
Mounts read-only after errors	ext4	Kernel remounted read-only after I/O or metadata errors	Unmount, e2fsck; check the disk (dmesg, SMART)
XFS: Log inconsistent, Log mount failed	xfs	Log damaged	Mount once for replay; xfs_repair; -L last

XFS: xfs_repair

XFS keeps a primary superblock in the first sector and a secondary superblock in every allocation group. xfs_repair can use a secondary when the primary is lost. It has no “force check at boot” step: XFS replays its log when you mount, and xfs_repair is a manual tool.

Option	Use
-n	No modify: report only (the first step, always)
(none)	Repair. Refuses a dirty log: mount and unmount once, to replay it
-L	Zero the log. Last resort: discards transactions not yet on disk
-v	Verbose

xfs_repair puts files it cannot attach back into lost+found named by inode number. A common outcome of serious damage is a lost+found full of numbered files and directories; recover what you can by inspecting them.

ext4: e2fsck

e2fsck (also called fsck.ext4) runs in five passes: inodes, directories, connectivity, reference counts, group summaries. Use -f to force a check of a file system that is marked clean, -n to answer “no” to every question (read-only), -y to answer “yes” (repair). ext4 keeps backup superblocks at the start of block groups. mke2fs -n -b BLOCKSIZE DEVICE prints their locations without writing anything, and e2fsck -b NUMBER uses one.

Try it

All outputs in this section are real, from xfs_repair and e2fsck on image files created with mkfs.xfs and mkfs.ext4 (the same commands work on a partition or LV).

XFS with a lost first sector

The first sector of an XFS image was overwritten with zeros.

mount -o loop x.img /mnt
blkid x.img; echo "blkid rc=$?"
xfs_repair -n x.img | head -10

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
blkid rc=2
Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

attempting to find secondary superblock...
.found candidate secondary superblock...
verified secondary superblock...
would write modified primary superblock
Primary superblock would have been modified.
Cannot proceed further in no_modify mode.
Exiting now.

blkid prints nothing and returns 2: it sees no file system. The no-modify run found a good secondary superblock, so a repair will work. Now repair:

xfs_repair x.img | tail -9
mount -o loop x.img /mnt && df -hT /mnt

Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
Note - stripe unit (0) and width (0) were copied from a backup superblock.
Please reset with mount -o sunit=<value>,swidth=<value> if necessary
done
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/loop0     xfs   236M   20M  217M   9% /mnt

The phases read as a story: find a superblock (1), check the log and inodes (2 to 4), rebuild free-space structures (5), reconnect orphans (6), fix link counts (7). The note about stripe unit matters only for RAID layouts: if the file system was created with su/sw on a striped array, reset them at mount time.

ext4 with a lost superblock

The first 4 KiB, and the primary superblock, were zeroed.

mke2fs -n -b 4096 e.img | tail -3
e2fsck -fn e.img | head -4

Filesystem UUID: 41b74a12-f9ab-47db-b151-c3de04538785
Superblock backups stored on blocks: 
	32768, 98304

e2fsck 1.47.0 (5-Feb-2023)
ext2fs_open2: Bad magic number in super-block
e2fsck: Superblock invalid, trying backup blocks...
Pass 1: Checking inodes, blocks, and sizes

e2fsck itself tried the backup. For the repair, point it at the backup explicitly:

e2fsck -fy -b 32768 e.img | tail -8
e2fsck -fn e.img | tail -2

Pass 5: Checking group summary information
Block bitmap differences:  +(32768--32818)
Fix? yes

Padding at end of inode bitmap is not set. Fix? yes


e.img: ***** FILE SYSTEM WAS MODIFIED *****
e.img: 11/102400 files (0.0% non-contiguous), 10663/102400 blocks
Pass 4: Checking reference counts
Pass 5: Checking group summary information
e.img: 11/102400 files (0.0% non-contiguous), 10663/102400 blocks

The “differences” are expected: backup superblocks hold slightly old summary counts, and e2fsck rewrites them. The second run (no repair) is clean. Always run a second check after a repair; it should say nothing.

A note on the numbers: this image was 400 MB with 4 KiB blocks, so the first backup is at 32768. A 100 MB file system has only one block group and no backup at all, so mke2fs -n lists none. The location depends on size and options, which is why you print it rather than remember it.

Hardware or file system?

Corruption can be a symptom, not the disease. If dmesg shows I/O error, Buffer I/O error or blk_update_request, the disk or path is failing: repairing the file system again will not help. Check SMART data (smartctl -a), cabling, and the virtualisation layer, copy off what you can first, and repair a copy.

Going further

Metadata dumps. xfs_metadump -g /dev/vdb1 meta.dump saves the metadata (not the data) of an XFS file system so a support engineer can examine its structure; it can be large. Beyond the exam: xfs_db inspects structures directly, xfs_admin -L label and -U uuid change labels, and XFS cannot be shrunk, only grown (xfs_growfs). ext4 can be shrunk only offline (resize2fs).