Which boot stage failed? Reading the evidence

The idea

A machine that “will not boot” has failed at one specific stage. The stage determines the tools, because each stage can reach only what it has loaded so far. Finding the stage first is half of the repair.

1  Firmware (UEFI/BIOS)   POST, finds a boot device, runs the boot loader (shim + GRUB on RHEL UEFI)
2  Boot loader (GRUB 2)   reads grub.cfg and the BLS entries, loads the kernel and initramfs
3  Kernel                 detects hardware, mounts the initramfs as a temporary root
4  initramfs (dracut)     assembles storage (LVM, LUKS, RAID), finds and mounts the real root, switches to it
5  systemd                starts units to reach default.target: mounts, networking, services, login

Stage	Where you see it fail	Where the evidence is
1 Firmware	No boot device, wrong order, Secure Boot rejects the loader	Firmware screen; not in any log
2 GRUB	grub> prompt, grub rescue>, “file not found”, missing entry	The screen only; repair from a rescue system
3 Kernel	Kernel panic, “Unable to mount root fs”, black screen after the kernel loads	Console text; on the next good boot journalctl -b -1 -k if it got that far
4 initramfs	dracut:/# emergency shell, “does not exist” for the root device, timeouts in dracut-initqueue	The shell itself: /run/initramfs/rdsosreport.txt, journalctl in the initramfs
5 systemd	“Welcome to emergency mode”, failed units, a long wait	The journal of that boot, readable on the next boot

Stages 1 to 4 happen before the real root file system is available, so there are no files to read unless you boot something else. Stage 5 writes a journal, and with a persistent journal you can read it later.

The journal that survives

By default (Storage=auto), systemd-journald keeps the journal in /run/log/journal, which disappears at reboot. It writes to /var/log/journal only if that directory exists. For troubleshooting boots, create it:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

After that journalctl --list-boots shows previous boots and journalctl -b -1 is the one before the current. Do this on a healthy machine, before the problem. A journal that did not exist cannot be read later.

Emergency and rescue

When stage 5 cannot complete, systemd falls back to a smaller target:

Target	What is running	Use
rescue.target	Base system, root mounted read-write, other mounts and services started where possible	Single-user repair when most things work
emergency.target	Only the root file system, mounted read-only, almost nothing else	When even rescue fails; remount with mount -o remount,rw / before editing

Both need the root password (or systemd.debug_shell-style methods) unless you reset it with rd.break. If you do not know the root password, see the initramfs lesson of this skill.

Try it

Read the previous boot

journalctl --list-boots
journalctl -b -1 -p err --no-pager | tail -5

IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
 -1 4c3b0e7a5f2a4d8b8f6f9b3c4b2f8a1e Wed 2026-11-04 09:10:41 +03 Wed 2026-11-04 09:14:02 +03
  0 b81e0d0c1e8e4f4c9a8c1c7d3e6a5f90 Wed 2026-11-04 09:15:12 +03 Wed 2026-11-04 10:30:55 +03
Nov 04 09:12:11 node1 systemd[1]: dev-disk-by\x2duuid-0000....device: Job dev-disk-by\x2duuid-0000....device/start timed out.
Nov 04 09:12:11 node1 systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-0000....device - /dev/disk/by-uuid/0000....
Nov 04 09:12:11 node1 systemd[1]: Dependency failed for mnt-ghost.mount - /mnt/ghost.

The three lines tell the whole story: a device the system waited for never appeared, the mount that needed it failed, and anything depending on that mount failed after it.

Where did the time go?

systemd-analyze reports the time spent in each stage and unit:

systemd-analyze
systemd-analyze blame | head -4
systemd-analyze critical-chain | head -8

Startup finished in 2.101s (kernel) + 4.882s (initrd) + 95.301s (userspace) = 1min 42.284s
graphical.target reached after 95.201s in userspace.
1min 30.003s dev-disk-by\x2duuid-0000....device
      6.211s NetworkManager-wait-online.service
      3.004s firewalld.service
graphical.target @1min 35.201s
└─multi-user.target @1min 35.200s
  └─mnt-ghost.mount @1min 30.100s +4ms
    └─dev-disk-by\x2duuid-0000....device @1min 30.100s

blame ranks units by their own time. critical-chain shows what actually delayed the target, which is not always the biggest unit: a long unit that runs in parallel costs nothing. Read critical-chain first, then blame.

Check fstab without rebooting

findmnt --verify is a dry-run of /etc/fstab. It reports unreachable devices, wrong file system types and bad options without mounting anything:

findmnt --verify

0 parse errors, 2 errors, 4 warnings
/data
   [E] unreachable on boot required target: No such file or directory
   [E] unreachable on boot required source: UUID=11111111-2222-3333-4444-555555555555
/srv
   [W] unreachable source: /dev/nonexistent: No such file or directory

This is real output from a test fstab with a UUID that does not exist. Run it after every edit of /etc/fstab, before you reboot. An [E] line on a non-nofail entry is a boot that will stop in emergency mode.

Going further

Reading an initramfs failure. If you are at the dracut:/# prompt, cat /run/initramfs/rdsosreport.txt has the dracut log and the state of the storage stack. journalctl works there too, and lsblk and blkid show what the kernel found. Beyond the exam: rd.debug and systemd.log_level=debug on the kernel command line produce very verbose logs of the initramfs and systemd; add them for one boot only, since the output is large.