Skill 2 in course 3.3: Software and Applications
Application Debugging
Trace system and library calls, find open files, ports and limits, read core dumps, and debug services under systemd.
You will be able to
- Use strace and ltrace to find the exact failing file, permission, network or library call behind a vague error.
- Find what a process holds open, which process owns a port, and which limit it hit.
- Collect a core dump with systemd-coredump and read its backtrace with gdb.
- Explain why a program works in a shell but not as a service, and reproduce the service environment.
Lessons
- 01Seeing what a program does with strace and ltraceTrace system calls and library calls to find the exact file, permission, connection or library behind a failing program.
stracestrace -fstrace -e trace=strace -cstrace -pstrace -o0/3 exercises - 02Open files, listening ports and resource limitsFind what a process holds open, who owns a port, and which limit or memory event stopped it.
lsoflsof -ilsof -pss -tulpnfuserls /proc/PID/fd0/3 exercises - 03Crashes: core dumps, coredumpctl and gdbCapture a crashed process with systemd-coredump, read its backtrace with gdb, and tell a crash from a clean error.
coredumpctlcoredumpctl listcoredumpctl infocoredumpctl debugcoredumpctl dumpgdb0/3 exercises - 04Works in my shell, fails as a serviceReproduce a service's real environment, and separate unit, permission, SELinux, network and sandbox causes.
systemctl catsystemctl showsystemd-runsystemd-run -tsystemd-analyze securityjournalctl -u0/3 exercises