Crashes: core dumps, coredumpctl and gdb

The idea

A program that dies from a signal such as SIGSEGV (it touched memory it does not own), SIGABRT (it called abort(), often after a failed assertion) or SIGBUS leaves the kernel the option to save a core dump: a copy of the process’s memory and registers at the moment of death. A core dump is not a log message; it is the frozen process. A debugger can then show which function was running, what called it, and the values of variables.

A crash and an error are different things. A program that prints an error and exits with status 1 has chosen to stop; the reason is in its output or its log. A program killed by a signal did not choose, the shell reports status 128 plus the signal number (139 for SIGSEGV), and systemctl status shows status=11/SEGV. Only the second kind produces a core dump.

How RHEL decides what happens to a dump

Four things must all permit it:

Gate	Where	Default on RHEL
Size limit of the process	ulimit -c, /proc/PID/limits, LimitCORE= in units	Shells usually have a soft limit of 0; for services read the live value with systemctl show UNIT -p LimitCORE
Handler	/proc/sys/kernel/core_pattern	|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h
Storage and caps	/etc/systemd/coredump.conf	Storage=external, ProcessSizeMax= and ExternalSizeMax= caps; compression on
Space	/var/lib/systemd/coredump/	Cleaned by systemd-tmpfiles after about three days

The leading | in core_pattern pipes the dump to a program instead of writing a core file in the working directory. systemd-coredump saves it as a compressed file, logs a journal entry with a short backtrace, and coredumpctl lets you query it.

coredumpctl list                       # recent crashes: time, PID, UID, signal, present?, size, program
coredumpctl info PROGRAM|PID           # details, including a stack trace if debug info is installed
coredumpctl debug PROGRAM|PID          # open the dump in gdb
coredumpctl dump PROGRAM -o core.file  # extract the core file itself

Reading a backtrace

The key tool is gdb:

gdb command	Meaning
bt	Backtrace: the chain of function calls at the crash, innermost (frame 0) first
frame N	Select a frame
list	Show source code around the line
info locals, info args	Variables in the selected frame
print EXPR, print *ptr	Evaluate an expression
info registers rip	The instruction that faulted

A backtrace shows symbols and source lines only when debug information is present. Compiling with -g provides it for your own code. For distribution packages, debug information is separate: dnf debuginfo-install PKG (repository *-debuginfo-rpms enabled), or gdb can fetch it on demand from Red Hat’s debuginfod server when DEBUGINFOD_URLS is set. Without it you still get function names for exported symbols, which is often enough to recognise a known bug in a package.

Try it

These outputs are real, from gdb 13 on a core file of a small C program with -g. (The machine in the test writes a plain core file; on RHEL the same file comes out of coredumpctl dump, and coredumpctl debug opens it for you.)

static int load(struct conf *c, const char *key) {
  if (strcmp(key, "name") == 0) { c->name = "labapp"; return 0; }
  return -1;           /* never sets c->name for other keys */
}
static void show(struct conf *c) { printf("%s:%d\n", c->name, c->port); }

The crash

./crash; echo "rc=$?"

Segmentation fault (core dumped)
rc=139

Status 139 is 128 + 11: killed by SIGSEGV. Now the dump:

gdb -batch -ex bt -ex 'print *c' -ex 'info registers rip' ./crash core

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000056031214720f in show (c=0x7ffc2143de60) at crash.c:8
#0  0x000056031214720f in show (c=0x7ffc2143de60) at crash.c:8
#1  0x000056031214727b in main () at crash.c:12
$1 = {name = 0x0, port = 0}
rip            0x56031214720f      0x56031214720f <show+60>

Read it from the top. Program terminated with signal SIGSEGV confirms the kind of failure. Frame 0 is show at crash.c:8, called from main at crash.c:12. The structure *c contains name = 0x0: a NULL pointer, and show used it in printf. The cause is upstream: load(&c, "port") returned -1 without setting name, and nobody checked the result. A fix is in the program, not in the system, but the diagnosis (and the evidence to report to the vendor or developer) came entirely from the dump.

On RHEL with coredumpctl

coredumpctl list --no-pager | tail -2
coredumpctl info crash --no-pager | sed -n 1,18p

TIME                          PID  UID  GID SIG     COREFILE EXE             SIZE
Wed 2026-11-04 12:21:09 +03  4410    0    0 SIGSEGV present  /tmp/crash     34.2K
           PID: 4410 (crash)
           UID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Wed 2026-11-04 12:21:09 +03 (12s ago)
  Command Line: ./crash
    Executable: /tmp/crash
       Storage: /var/lib/systemd/coredump/core.crash.0.....zst (present)
       Message: Process 4410 (crash) of user 0 dumped core.
                Stack trace of thread 4410:
                #0  0x000055f1... show (/tmp/crash + 0x120f)
                #1  0x000055f1... main (/tmp/crash + 0x127b)

(Representative output.) The column COREFILE says present while the file is still on disk and missing once it has been cleaned up. If you need a dump later, copy it: coredumpctl dump crash -o /root/crash.core.

A crash is also a service event

Because the kernel hands the dump to systemd, the crash appears in the journal even if the application kept no log:

journalctl -u myapp.service -b --no-pager | tail
systemctl status myapp.service

myapp.service: Main process exited, code=dumped, status=11/SEGV
myapp.service: Failed with result 'core-dump'.

code=dumped and Result: core-dump mark a crash. With Restart=on-failure the service will start again; repeated crashes show as a loop of these lines, so count them (coredumpctl list shows how many).

When there is no dump

Cause	Check	Fix
Core limit is 0	grep core /proc/PID/limits	LimitCORE=infinity in a unit drop-in; ulimit -c unlimited in a shell
Handler is not systemd-coredump	cat /proc/sys/kernel/core_pattern	Install the package providing it and restart; do not point it at a world-writable path
Dump larger than the caps	/etc/systemd/coredump.conf	Raise ProcessSizeMax= and ExternalSizeMax=
Disk full	df -h /var	Free space; coredumpctl also has a KeepFree= setting
Privilege-dropping or setuid program	sysctl fs.suid_dumpable	Value 2 allows dumps, but protects them from ordinary users
The process was killed by SIGKILL	status=9/KILL	SIGKILL produces no dump; look for the OOM killer or a human
Container or PID namespace		coredumpctl works on the host for host-visible processes

Going further

Sensitive data. A core dump contains everything the process had in memory: keys, passwords, session data. Treat dumps like secrets. They are stored root-only, and Storage=none in coredump.conf turns storage off for machines where that is unacceptable. Beyond the exam: gdb can attach to a live process (gdb -p PID) to take a backtrace without stopping the service permanently (gcore PID writes a core of a running process), valgrind finds invalid memory use before it becomes a crash, and abrt may be installed to report crashes to Red Hat automatically on systems that use it.