Capturing a kernel crash with kdump

The idea

When the kernel itself fails (a panic, or a hard hang you trigger), nothing running on that kernel can save evidence, because the kernel is the thing that has failed. kdump solves this with a second kernel waiting in reserved memory.

Boot:   the normal kernel reserves a block of RAM (crashkernel=) and loads a small
        "capture kernel" and its initramfs into it. That RAM is invisible to the normal kernel.
Panic:  the normal kernel jumps (kexec) into the capture kernel, without a BIOS reset,
        so the contents of RAM are untouched.
Capture: the capture kernel starts, copies the old kernel's memory (/proc/vmcore) to the
        dump target with makedumpfile, then reboots into a normal boot.

The memory must be reserved at boot, from the kernel command line. A running system cannot reserve it, which is why changing crashkernel= always needs a reboot.

The pieces

Piece	Where	Notes
Package	kexec-tools	Provides kdump.service, kdumpctl and /etc/kdump.conf
Reservation	crashkernel= on the kernel command line	Size depends on RAM; RHEL sets a default range at install
Configuration	/etc/kdump.conf	Dump target (path, or nfs, ssh, a device), collector, failure action
Service	kdump.service	Builds the capture initramfs and loads the capture kernel
Check flag	/sys/kernel/kexec_crash_loaded	1 when the capture kernel is loaded
Result	/var/crash/HOST-DATE/vmcore	The memory image; vmcore-dmesg.txt is the kernel log extracted from it

The dump target needs free space. makedumpfile shrinks the image by dropping zero pages and, with -d 31, cache and free pages, but a busy server’s dump can still be several gigabytes. kdumpctl estimate reports the expected size and a suggested reservation.

Configuration in /etc/kdump.conf

Most lines are comments. The ones to know:

path /var/crash
core_collector makedumpfile -l --message-level 7 -d 31

path is the directory under the dump target (the default target is the local root file system). core_collector is the command that copies and compresses the image (-l for LZO, -d 31 excludes the page types that are useless for most analysis). Other targets exist (nfs host:/export, ssh user@host, ext4 /dev/vgdata/dumps); changing the target requires systemctl restart kdump so the capture initramfs is rebuilt.

When you change /etc/kdump.conf, restart the service. The capture initramfs is built from the file at that moment; it does not follow later edits.

Reserving memory

On a new RHEL 10 installation the installer normally sets a default reservation. Check:

grep -o 'crashkernel=[^ ]*' /proc/cmdline

If it is empty, set it for all installed kernels and reboot:

sudo kdumpctl reset-crashkernel --kernel=ALL
sudo grubby --info=DEFAULT | grep args

reset-crashkernel applies the default rule for the machine’s memory. To choose a value yourself, use grubby --update-kernel=ALL --args="crashkernel=256M" and check kdumpctl estimate for a safe size. Too small a reservation makes the capture kernel run out of memory while it is writing, and no dump appears: the most common reason a kdump test “does nothing”.

Try it

grep -o 'crashkernel=[^ ]*' /proc/cmdline
systemctl is-enabled kdump; systemctl is-active kdump
cat /sys/kernel/kexec_crash_loaded
sudo kdumpctl status

crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
enabled
active
1
kdump: Kdump is operational

If kdumpctl status says the service is not operational, read the log:

journalctl -u kdump -b --no-pager | tail -5

kdump[812]: No memory reserved for crash kernel
kdump[812]: Starting kdump: [FAILED]

Testing the capture

A real test is the only proof. Use a lab machine, because the machine reboots without a clean shutdown.

echo 1 | sudo tee /proc/sys/kernel/sysrq      # allow SysRq commands
echo c | sudo tee /proc/sysrq-trigger         # crash the kernel

The console shows a panic, the capture kernel boots, writes the dump and the machine restarts. When it is back:

ls -lh /var/crash/*/

-rw-------. 1 root root 331M Nov  4 11:42 vmcore
-rw-r--r--. 1 root root  70K Nov  4 11:42 vmcore-dmesg.txt
-rw-r--r--. 1 root root  18K Nov  4 11:42 kexec-dmesg.log

Reading the first clue

You do not need a debugger to start. vmcore-dmesg.txt is the kernel log from the moment of the crash, and the last lines usually name the reason.

tail -15 /var/crash/*/vmcore-dmesg.txt

[  412.204811] sysrq: Trigger a crash
[  412.204816] Kernel panic - not syncing: sysrq triggered crash
[  412.204821] CPU: 1 PID: 4311 Comm: tee Kdump: loaded Not tainted 6.12.0-55.el10.x86_64 #1
...

Look for: Kernel panic - not syncing: followed by the reason; BUG: or Oops: lines above it with a function name; Out of memory: (OOM with panic_on_oom); Call Trace: listing the functions on the stack. A tainted flag in the CPU line tells you whether third-party or forced modules were loaded.

For deeper analysis the crash utility opens the vmcore together with the matching debug kernel (package kernel-debuginfo, which comes from the debug repository, so it may not be available offline):

crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/*/vmcore

Inside crash, bt shows the backtrace of the panicking task, log the kernel buffer, ps the tasks, kmem -i memory usage. Read the dump on a copy, not on production, and expect support to ask for the vmcore, which may contain sensitive data from memory.

When no dump appears

Symptom	Cause	Fix
No memory reserved for crash kernel	No crashkernel=	kdumpctl reset-crashkernel or grubby ... --args, reboot
Service active, crash gives plain reboot	Capture kernel not loaded or reservation too small	Check kexec_crash_loaded; raise the reservation
Dump directory empty	Target full or path wrong, or capture kernel ran out of memory	df -h /var/crash; read kexec-dmesg.log on the next boot
Service starts, then fails after a config edit	Typo in /etc/kdump.conf	journalctl -u kdump; fix, systemctl restart kdump
Works on one kernel, not after an update	Reservation or initramfs for the new kernel	kdumpctl restart; reset-crashkernel --kernel=ALL
SysRq does nothing	kernel.sysrq is 0	echo 1 > /proc/sys/kernel/sysrq

Going further

Production notes. Keep /var/crash on a file system with room for at least the RAM size on a large machine, or send dumps over NFS or SSH. Set the failure action (failure_action reboot is the default) so a failed capture does not leave the host hanging. Beyond the exam: kernel.panic_on_oops, kernel.hung_task_panic and the watchdog decide which failures produce a panic, and therefore a dump. A hang that never panics produces nothing unless you trigger it.