The initramfs, rd.break and recovering root access

The idea

The kernel cannot mount the real root file system until it can read the device that holds it. That device may sit on LVM, behind LUKS encryption, on RAID, on NVMe, or on a network. None of that code is built into the kernel in a distribution kernel, so the boot loader hands the kernel a small helper root file system, the initramfs (“initial RAM file system”), which contains just enough to assemble storage, find the real root and switch to it.

On RHEL the initramfs is generated by dracut:

kernel starts  ->  unpacks initramfs into RAM  ->  runs systemd (a small copy)
   -> loads storage drivers  -> activates LVM / unlocks LUKS / assembles RAID
   -> mounts root at /sysroot  ->  switches root to /sysroot  ->  real systemd starts

A failure here shows as a dracut message, a timeout in dracut-initqueue, and an emergency shell with the prompt dracut:/#, or switch_root:/# at a break point.

What is inside, and why it matters

Content	Why
Kernel modules for disk controllers, file systems (xfs), device mapper	Without the module, the device or file system does not exist
lvm, cryptsetup, mdadm tools and config	To activate the storage stack
/etc/fstab (not included)	Only the root device is known, from the kernel command line (root=, rd.lvm.lv=)
A copy of /etc/crypttab, /etc/lvm, /etc/mdadm.conf when needed	The initramfs has its own snapshot, taken when it was built

The last row is the source of many surprises. The initramfs knows about storage as it was at build time. If you change root storage later (new VG name, new UUID, move to a different controller) and do not rebuild it, the next boot looks for the old layout and fails. That is why you run dracut -f after such changes.

Tools

lsinitrd                       # contents of the default kernel's image (also: lsinitrd IMAGE)
lsinitrd -f etc/fstab          # print one file inside it
sudo dracut -f                 # rebuild for the running kernel, overwrite
sudo dracut -f --kver 6.12.0-55.el10.x86_64

Configuration lives in /etc/dracut.conf.d/*.conf; the /usr/lib/dracut/dracut.conf.d/ files belong to packages. Typical lines: add_drivers+=" nvme ", omit_dracutmodules+=" plymouth ", hostonly="yes" (include only what this machine needs; the default) or hostonly="no" (a generic image that boots anywhere, larger).

Breaking in on purpose: rd.break

rd.break stops the initramfs after it has found the real root and before it switches to it. You get a shell with root privileges and no login, because nothing from the real system is running. That is why anybody with console access and the ability to edit the GRUB line can set the root password, and why the GRUB menu and the machine room door are part of the security model.

At the prompt:

Step	Command	Why
1	mount -o remount,rw /sysroot	The real root is mounted read-only here
2	chroot /sysroot	Make the installed system your / so tools and /etc/shadow are the real ones
3	passwd root	Change the password
4	touch /.autorelabel	Request a full SELinux relabel on the next boot
5	exit, exit	Leave the chroot, then continue the boot

Step 4 is not optional. The shell you are in has no SELinux policy loaded, so the new /etc/shadow that passwd writes gets no proper label. With the default enforcing mode, the next login can fail with “Authentication failure” even though the password is right. The relabel fixes the label; it takes minutes on a big file system and the console shows the progress. You can instead run restorecon /etc/shadow inside the chroot, but .autorelabel is the dependable habit.

Try it

sudo lsinitrd | sed -n 1,6p
sudo lsinitrd | grep -E 'xfs.ko|bin/lvm'

Image: /boot/initramfs-6.12.0-55.el10.x86_64.img: 34M
========================================================================
Early CPIO image
========================================================================
drwxr-xr-x   3 root     root            0 Nov  4 10:15 .
-rw-r--r--   1 root     root            2 Nov  4 10:15 early_cpio
usr/lib/modules/6.12.0-55.el10.x86_64/kernel/fs/xfs/xfs.ko.xz
usr/sbin/lvm

echo 'add_drivers+=" nvme "' | sudo tee /etc/dracut.conf.d/90-nvme.conf
sudo cp -p /boot/initramfs-$(uname -r).img{,.bak}
sudo dracut -f --kver $(uname -r)
sudo lsinitrd | grep nvme | head -3

If the new image fails to boot, choose another kernel in the GRUB menu, boot, and restore the copy (cp -p ...img.bak ...img). That is why you keep a .bak until a reboot succeeds.

Recovering from a missing root device

At a dracut:/# prompt after a timeout, the evidence is in the shell itself:

cat /run/initramfs/rdsosreport.txt | less      # the log and a storage summary
lsblk; blkid                                    # what the kernel found
cat /proc/cmdline                               # root=, rd.lvm.lv=, rd.luks.uuid=
lvm vgscan; lvm vgchange -ay                    # can LVM see the VG?

Typical causes and their repairs:

rdsosreport or screen says	Meaning	Repair
Warning: /dev/mapper/rhel-root does not exist and no disks in lsblk	Storage driver or controller missing in the image	Boot the old kernel, add the driver, dracut -f
Disk visible, lvm vgscan finds nothing	LVM filter or devices file excludes the disk	Boot rescue, fix /etc/lvm/devices/system.devices, dracut -f
rd.lvm.lv=oldvg/root in /proc/cmdline but VG is now newvg	Renamed VG, stale command line	Edit the GRUB line for one boot, then grubby --update-kernel=ALL --args=... --remove-args=... and dracut -f
UUID in root= not found	Root file system re-created with a new UUID	Same, with the new UUID from blkid
Encrypted root: no passphrase prompt	rd.luks.* argument or crypttab copy missing	Rebuild with the crypt module and correct /etc/crypttab

Going further

Rescue media or rd.break? If you can edit GRUB, rd.break is the shortest path to a shell. If you cannot (GRUB password, no console access), the installation media’s rescue mode gives the same result with the installed system mounted under /mnt/sysroot. Beyond the exam: rd.shell, rd.debug and rd.retry control how the initramfs behaves when it fails, and systemd.debug_shell adds a root shell on tty9 during a normal boot, a debugging aid you must never leave enabled.