2 of 4

Startup and Boot FailuresLesson 2 of 4

GRUB, boot entries and kernel arguments

Edit kernel arguments for one boot, make them permanent with grubby, and rebuild a missing or broken boot configuration.

  • grubby
  • grubby --info=ALL
  • grubby --update-kernel
  • grubby --set-default
  • grub2-mkconfig
  • grub2-editenv
  • /etc/default/grub
  • /boot/loader/entries
  • /proc/cmdline
  • dnf reinstall
  • kernel-install

50 min3 exercises

After this lesson you can

  • Edit the kernel command line for a single boot from the GRUB menu.
  • Add, remove and verify permanent kernel arguments, and choose the default entry, with grubby.
  • Rebuild grub.cfg and restore a missing boot entry or kernel from a running or rescue system.

The idea

GRUB 2 starts the kernel. On RHEL it needs three things: its own files, a list of boot entries, and a place to remember the default. How those are kept has changed over the years, and mixing old and new habits is the usual source of trouble.

Where the pieces are

ItemLocationNotes
Boot loader (UEFI)/boot/efi/EFI/redhat/shimx64.efi, grubx64.efi, grub.cfg (a short stub on UEFI)
Boot loader (BIOS)In the MBR/BIOS-boot partition, files in /boot/grub2/grub.cfg here is the real configuration
Boot entries/boot/loader/entries/*.confBLS (Boot Loader Specification): one file per kernel, with title, linux, initrd, options
Default entry/boot/grub2/grubenv (saved_entry)Edited by grub2-editenv or grubby --set-default
Settings/etc/default/grubDefaults used when regenerating; GRUB_CMDLINE_LINUX is the shared kernel argument line
Kernels/boot/vmlinuz-*, /boot/initramfs-*.imgInstalled by the kernel-core package; /boot is often its own partition

A BLS entry looks like this:

title Red Hat Enterprise Linux (6.12.0-55.el10.x86_64) 10.2
version 6.12.0-55.el10.x86_64
linux /vmlinuz-6.12.0-55.el10.x86_64
initrd /initramfs-6.12.0-55.el10.x86_64.img $tuned_initrd
options root=/dev/mapper/rhel-root ro crashkernel=1G-4G:192M rd.lvm.lv=rhel/root rhgb quiet $tuned_params
grub_users $grub_users
grub_arg --unrestricted
grub_class rhel

The options line is the kernel command line. grubby edits this file; grub2-mkconfig does not need to be run for a normal argument change on a current RHEL.

/etc/default/grub versus grubby

  • GRUB_CMDLINE_LINUX in /etc/default/grub is the argument line used when new kernels are installed. Changing it and running grub2-mkconfig does not alter existing entries on a BLS system.
  • grubby --update-kernel=ALL --args=... changes the entries that already exist. To affect future kernels as well, update both: grubby for what exists now, /etc/default/grub for what will be installed.

What the GRUB prompts mean

You seeMeaningFirst action
Menu, then kernel loads normallyFine
Menu has no entry for the kernel you needEntry file missing or /boot not readableBoot another entry, or rescue; then kernel-install add
grub>grub.cfg not found or unreadablels, ls (hd0,gpt2)/ to find the files; configfile or boot manually
grub rescue>GRUB lost even its modulesBoot rescue media, chroot, reinstall (below)
error: unknown filesystem/boot file system damaged or driver missingRescue, xfs_repair/e2fsck on /boot

Try it

For one boot: edit the entry in the menu

At the GRUB menu press e on the entry. Find the line that starts with linux, move to its end, and add the argument. Ctrl+x boots with it; Esc abandons the edit. Nothing is saved.

Common one-boot arguments:

AddEffect
systemd.unit=rescue.targetBoot to rescue mode
systemd.unit=emergency.targetBoot to emergency mode
rd.breakStop in the initramfs before the switch to the real root
enforcing=0SELinux permissive for this boot (diagnosis only)
init=/bin/bashStart a shell instead of systemd (read-only root, no services)
systemd.mask=foo.serviceDo not start one unit
Remove rhgb quietShow boot messages

Permanently: grubby

sudo grubby --info=DEFAULT
index=0
kernel="/boot/vmlinuz-6.12.0-55.el10.x86_64"
args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.lvm.lv=rhel/root rhgb quiet"
root="/dev/mapper/rhel-root"
initrd="/boot/initramfs-6.12.0-55.el10.x86_64.img"
title="Red Hat Enterprise Linux (6.12.0-55.el10.x86_64) 10.2"
id="6fa8...-6.12.0-55.el10.x86_64"
sudo grubby --update-kernel=ALL --args="audit=1" --remove-args="rhgb quiet"
sudo grubby --info=ALL | grep ^args
sudo grubby --default-kernel
sudo grubby --set-default /boot/vmlinuz-6.12.0-55.el10.x86_64

--update-kernel accepts ALL, DEFAULT or a path. --args adds or changes, --remove-args removes. After a reboot, confirm in /proc/cmdline, because that is what the kernel actually received, not what the file says.

Rebuilding the configuration

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

On UEFI systems RHEL’s grub.cfg in /boot/efi/EFI/redhat/ is a stub that loads /boot/grub2/grub.cfg, so the command above is correct on both firmware types on current releases; if the output path is wrong for your layout, the man page grub2-mkconfig(8) and the existing grub.cfg stub show it.

Bringing back a boot entry or the boot loader

From a running system, or from rescue media after you mount the installed system and chroot into it:

# entries
sudo kernel-install add $(uname -r) /lib/modules/$(uname -r)/vmlinuz
# missing kernel or initramfs files
sudo dnf reinstall kernel-core
sudo dracut -f --kver $(uname -r)
# damaged UEFI boot files
sudo dnf reinstall shim-x64 grub2-efi-x64
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

On UEFI you do not run grub2-install on RHEL; reinstalling the shim and grub2-efi packages restores the files in the EFI system partition. If the firmware no longer lists the entry, efibootmgr -v shows the boot variables and efibootmgr -c recreates one. On BIOS systems grub2-install /dev/sda rewrites the loader to the disk.

The rescue steps, in order, when the installed system will not start:

  1. Boot the installation media, choose Troubleshooting → Rescue a Red Hat Enterprise Linux system.
  2. Let it find the installation (1 Continue mounts it at /mnt/sysroot, read-write).
  3. chroot /mnt/sysroot, repair, exit, reboot.
  4. If you changed files in a SELinux enforcing system, run touch /.autorelabel before rebooting.

When the boot loader is the problem

SymptomCauseFix
Menu has old kernels onlyNew kernel’s entry missingkernel-install add or reinstall kernel-core
New argument not in /proc/cmdlineEdited /etc/default/grub onlygrubby --update-kernel=ALL --args=...
Argument present after update, but gone after a new kernel installsOnly the entries were changedUpdate GRUB_CMDLINE_LINUX as well
Wrong kernel boots by defaultsaved_entry points elsewheregrubby --set-default
grub> promptgrub.cfg unreachablels/configfile to boot once, then repair
Kernel loads, then “Unable to mount root”Wrong root= or missing driverCompare root= with lsblk; rebuild initramfs
Boot fails only with Secure Boot onUnsigned module or loaderUse the signed packages; do not copy loaders by hand

Going further

Protecting the menu. Editing the menu with e is the reason the physical console must be trusted. A GRUB password (grub2-setpassword) requires it for edits while still allowing the default boot. Beyond the exam: UEFI boot variables, signed kernels with Secure Boot, and the Unified Kernel Image format are all moving how boot is configured; RHEL 10 still uses the model above.

Practice

0 of 3 done

Do these on your own system. They get harder. Check each result, then tick it off.

  1. Warm-up

    One boot only

    Reboot and, from the GRUB menu, boot the default kernel into rescue mode for this boot only, without changing any file. After login, prove the change is temporary: show the argument in /proc/cmdline, then reboot normally and show it is gone.

    Check your result
    • In the rescue boot, grep -o 'systemd.unit=rescue.target' /proc/cmdline prints the argument and systemctl get-default still prints multi-user.target or graphical.target.
    • After a normal reboot, the same grep prints nothing.
    Hint

    In the GRUB menu press e, go to the line starting with linux, press End, append systemd.unit=rescue.target, then Ctrl+x to boot. The edit exists only in memory.

    Solution
    GRUB menu: e  ->  cursor to the end of the "linux" line  ->  add:  systemd.unit=rescue.target  ->  Ctrl+x
    grep -o 'systemd.unit=[^ ]*' /proc/cmdline
systemctl get-default
sudo systemctl reboot
grep -o 'systemd.unit=[^ ]*' /proc/cmdline || echo "gone"
  2. Practice

    Permanent arguments with grubby

    For all installed kernels: add audit=1 and loglevel=5, remove rhgb and quiet, and make sure there is no crashkernel= change. Show the resulting arguments from grubby, reboot, and confirm the running kernel has them. Then set the default boot entry to the kernel you booted and show it.

    Check your result
    • sudo grubby --info=ALL | grep ^args contains audit=1 loglevel=5 and not rhgb or quiet.
    • After the reboot, grep -oE 'audit=1|loglevel=5' /proc/cmdline prints both; rhgb and quiet are absent.
    • sudo grubby --default-kernel prints the path of the running kernel (/boot/vmlinuz-$(uname -r)).
    Hint

    grubby --update-kernel=ALL --args="a=1 b=2" --remove-args="rhgb quiet". Do not edit grub.cfg by hand on RHEL: the boot entries live in /boot/loader/entries/*.conf and the kernel arguments in the options line.

    Solution
    sudo grubby --update-kernel=ALL --args="audit=1 loglevel=5" --remove-args="rhgb quiet"
sudo grubby --info=ALL | grep -E '^(index|args)'
sudo grubby --set-default /boot/vmlinuz-$(uname -r)
sudo grubby --default-kernel
sudo systemctl reboot
# after the reboot
cat /proc/cmdline
  3. Challenge

    The kernel entry disappeared

    Setup (lab VM):

    sudo mkdir -p /root/entries.bak
sudo mv /boot/loader/entries/*$(uname -r)*.conf /root/entries.bak/
sudo grubby --info=ALL | grep -c '^kernel'

    grubby now shows no usable entry for the running kernel and a reboot would land on the wrong one or on a grub> prompt. Without restoring the file from /root/entries.bak, recreate a correct boot entry from the installed kernel package, make it the default, and prove it with grubby --info=ALL and a reboot.

    Check your result
    • ls /boot/loader/entries/ contains an entry for $(uname -r).
    • sudo grubby --default-kernel prints /boot/vmlinuz-$(uname -r).
    • After a reboot uname -r is the expected version.
    Hint

    Entries are created by kernel-install when a kernel package is installed. Reinstalling the kernel core package re-runs it, or call kernel-install add VERSION /lib/modules/VERSION/vmlinuz directly. Then regenerate the config only if grubby says it is needed.

    Solution
    sudo kernel-install add $(uname -r) /lib/modules/$(uname -r)/vmlinuz
ls /boot/loader/entries/
sudo grubby --info=ALL | grep -E '^(index|kernel|args)'
sudo grubby --set-default /boot/vmlinuz-$(uname -r)
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo systemctl reboot