A method for troubleshooting, and answers from the machine itself

The idea

Troubleshooting is not a list of commands. It is a loop you run until the system behaves, and the quality of the loop decides how fast you finish and how little you break.

1. Observe     what exactly fails? since when? for whom? what changed?
2. Hypothesise one specific cause that would explain every symptom
3. Test        a read-only check that would prove the cause wrong
4. Change      ONE thing, and remember how to undo it
5. Verify      repeat the original failing action, then reboot-proof it
6. Record      write down symptom, cause, fix and proof

Three habits make the loop work.

Be precise about the symptom. “Web server is down” hides four different faults. “curl localhost is refused, systemctl says failed, exit code 203” points at one layer. Always reproduce the failure yourself before you theorise, and keep the exact command that fails, because it is also your test for step 5.

Work from the bottom up, and from the evidence. The system already wrote down most failures. Read the log of the thing that failed first, then the thing it depends on. A service fails because of its unit file, its binary, its permissions, its SELinux label, its port, its configuration or its dependencies, in roughly that order of likelihood.

Change one thing at a time. Two changes at once teach you nothing about which one fixed it, and a wrong change you cannot undo is a new fault. Take a copy (cp -a file file.orig), edit, test, and keep going from a known state.

First evidence

Four read-only questions answer most of what you need to start.

Question	Command	What it tells you
Is the thing running, and how did it last end?	systemctl status UNIT	State, main PID, exit status, last log lines
What did it log?	journalctl -u UNIT -b -p warning	Messages from this boot, warnings and worse
What did the kernel say?	dmesg --level=err,warn	Hardware, file systems, OOM killer, SELinux/audit lines
What else is broken?	systemctl --failed	A failed dependency often explains the symptom

Then widen: journalctl -b -1 (the previous boot, if the journal is persistent), ausearch -m avc -ts recent (SELinux), ss -tulpn (who listens), rpm -V PACKAGE (did a file change), df -h; df -i (full disk or inodes).

Exit codes you will see in systemctl status

Status	Meaning	Usual cause
203/EXEC	systemd could not execute ExecStart	Wrong path, no execute bit, SELinux label, missing interpreter
200/CHDIR, 217/USER	Bad WorkingDirectory= or User=	Directory or account does not exist
1/FAILURE or 2	The program started and exited with an error	Read the program’s own log
status=11/SEGV, 6/ABRT	The program crashed	A core dump or a bug: see the application debugging skill
Result: timeout	Type= or start timeout wrong	A forking daemon declared as simple, or a stuck start

Try it

systemctl status reportd --no-pager

× reportd.service - Report writer
     Loaded: loaded (/etc/systemd/system/reportd.service; disabled; preset: disabled)
     Active: failed (Result: exit-code) since Tue 2026-11-04 10:12:31 +03; 6s ago
    Process: 4120 ExecStart=/usr/local/bin/reportd --interval 30 (code=exited, status=203/EXEC)
   Main PID: 4120 (code=exited, status=203/EXEC)

Nov 04 10:12:31 node1 systemd[1]: Started reportd.service - Report writer.
Nov 04 10:12:31 node1 systemd[4120]: reportd.service: Failed at step EXEC spawning /usr/local/bin/reportd: Permission denied
Nov 04 10:12:31 node1 systemd[1]: reportd.service: Main process exited, code=exited, status=203/EXEC

The three useful facts are 203/EXEC, Failed at step EXEC and Permission denied. Together they say “systemd found the file but may not run it”. That is a hypothesis you can test with one read-only command:

ls -lZ /usr/local/bin/reportd

-rw-------. 1 root root unconfined_u:object_r:bin_t:s0 79 Nov  4 10:11 /usr/local/bin/reportd

The mode 600 has no execute bit, and the SELinux type bin_t is the normal type for programs in /usr/local/bin, so the label is not the problem. Reading the label before blaming SELinux is the point of testing a hypothesis: you rule a cause in or out in one command. Fix the mode, start the unit again and read the new status. If the label had been wrong (a file moved in from a home directory keeps user_home_t), the next message would say so and you would fix that second, not at the same time.

Documentation you already have

There is no internet on the exam, but the machine documents itself.

You want	Use
A command’s options	man CMD, CMD --help, info CMD
A configuration file’s syntax	man 5 FILE, for example man 5 sshd_config, man 5 fstab
“What is the command for …?”	man -k KEYWORD (same as apropos). Run sudo mandb if the index is empty
Which section?	man -f passwd lists passwd(1) and passwd(5); man 5 passwd picks one
Examples and templates shipped with a package	rpm -qd PACKAGE, then read the files under /usr/share/doc
Config files of a package, and whether you changed them	rpm -qc PACKAGE, rpm -V PACKAGE
Which package owns a file	rpm -qf /path
What a unit really runs, with drop-ins	systemctl cat UNIT
All directives of a unit file	man systemd.service, man systemd.exec, man systemd.unit
Ansible module syntax	ansible-doc ansible.builtin.lineinfile
SELinux per-service help	man httpd_selinux (needs selinux-policy-doc; run sudo mandb after installing)

rpm -qd chrony

/usr/share/doc/chrony/FAQ
/usr/share/doc/chrony/NEWS
/usr/share/doc/chrony/README
/usr/share/man/man1/chronyc.1.gz
/usr/share/man/man5/chrony.conf.5.gz

Going further

What to record. A useful note has four parts: the symptom as a command that fails, the cause, the change you made and the command that now succeeds. Writing it takes a minute and lets you undo it later; it is also the habit that support engineers expect when you hand a case to someone else (see the Crash Dumps and Support Data skill).

Beyond the exam. Real incidents add one question the exam never asks: what changed? Package history (dnf history), file changes (rpm -Va, AIDE), unit overrides (systemd-delta) and the time of the first failure in the journal usually point to the change. Ask it first in production.