Reading logs and keeping the journal

The idea

On RHEL the core log is the systemd journal, written by systemd-journald. It is a binary database that records every unit’s output, the kernel, and anything sent with logger, tagged with fields such as the unit, PID, priority and boot. You query it with journalctl instead of reading text files, which makes precise questions possible.

On systems that also run rsyslog, the traditional text logs exist as well: /var/log/messages (general), /var/log/secure (authentication), /var/log/cron, /var/log/maillog. The journal is always there; check whether the text files exist with ls /var/log before assuming they do.

Filters you will use all the time

Question	Command
Messages from one service	journalctl -u sshd
Only errors and worse	journalctl -p err (levels: emerg alert crit err warning notice info debug)
This boot / previous boot	journalctl -b / journalctl -b -1
A time window	journalctl --since "1 hour ago" --until "10 min ago"
Follow live	journalctl -f (combine with -u)
Kernel messages	journalctl -k
Text search	journalctl -g "denied"
By tag, process or PID	journalctl -t NAME, _COMM=sudo, _PID=1234
Newest first / last N	journalctl -r, journalctl -n 20
Show all fields	journalctl -o verbose

Filters combine: journalctl -u httpd -p err -b --since today is “errors from Apache since this boot that happened today”.

Persistent or not?

By default the journal is stored under /run/log/journal, which is in memory and disappears at reboot. It becomes persistent (kept in /var/log/journal on disk) when that directory exists, or when Storage=persistent is set in /etc/systemd/journald.conf. The exam phrase “preserve the system journal across reboots” means exactly this. Do the directory method and verify with journalctl --list-boots after a reboot.

Try it

journalctl -u sshd -n 3 --no-pager
journalctl -p err -b --no-pager | head -3

Nov 03 08:01:12 rhel10 sshd[842]: Server listening on 0.0.0.0 port 22.
Nov 03 08:01:12 rhel10 sshd[842]: Server listening on :: port 22.
Nov 03 09:32:50 rhel10 sshd[3011]: Accepted publickey for alex from 192.168.122.1 port 51744 ssh2
Nov 03 08:01:20 rhel10 systemd[1]: Failed to start tuned.service - Dynamic System Tuning Daemon.

Write your own entries

logger -t labtest -p user.warning "EX200 lab test"
journalctl -t labtest --no-pager

Nov 03 11:30:05 rhel10 labtest[5120]: EX200 lab test

This is how a cron job from the scheduling lessons can leave evidence, and how you test that logging works.

A time window and live following

journalctl --since "20 min ago" --no-pager | tail -2
journalctl -u httpd -f

-f keeps running and prints new lines; press Ctrl+C to stop. It is the first thing to open in a second terminal while you restart a failing service.

Make it persistent

ls -d /var/log/journal /run/log/journal

ls: cannot access '/var/log/journal': No such file or directory
/run/log/journal

Only the in-memory copy exists. Create the directory and restart the daemon:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush
journalctl --list-boots

IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
  0 2fbb9b3e0c0a4e1a9d7c5b2e1c6a1f10 Tue 2026-11-03 08:01:05 +03 Tue 2026-11-03 11:35:41 +03

After a reboot a second line appears (with index -1), which proves the old boot was kept. To state the choice in the configuration instead:

[Journal]
Storage=persistent

Keep it from filling the disk

journalctl --disk-usage
sudo journalctl --vacuum-size=100M

Archived and active journals take up 48.0M in the file system.
Vacuuming done, freed 0B of archived journals from /var/log/journal.

A persistent journal otherwise grows until it uses up to 10 % of the file system (and no more than 4 GB). Cap it with SystemMaxUse=200M in the [Journal] section, or remove old entries with --vacuum-time=2weeks, then restart systemd-journald.

Going further

journalctl -o json-pretty produces machine-readable output; -o cat strips everything except the message, handy in scripts. rsyslog rules in /etc/rsyslog.d/*.conf route messages to files by facility and priority (local0.* /var/log/lab.log), and logrotate (config in /etc/logrotate.conf and /etc/logrotate.d/, run daily by a timer) rotates those text files. journalctl --verify checks journal integrity. In RHEL 10 check rpm -q rsyslog before relying on /var/log/messages. Audit messages from SELinux go to /var/log/audit/audit.log, covered in the SELinux skill.