Part A: Boot, storage and evidence (100 points)

Do these on your own system. They get harder. Check each result, then tick it off.

1. Warm-upDone

   A1 · Recover node1 from emergency mode (12 points, 10 min)

   After the setup, node1 boots into emergency mode. Log in with the root password and:

   1. Find out which entry stopped the boot, and why.
   2. Repair /etc/fstab so that the machine reaches the default target. Do not delete lines that belong to working file systems.
   3. The data disk that the failed line referred to is not installed: if the file system is optional, boot must not wait for it for more than 5 seconds; if it does not exist, remove the line.
   4. Continue to the normal target without reinstalling anything.
   Check your result

   • systemctl is-system-running prints running (or degraded with no failed mount units).
   • findmnt --verify prints no [E] lines.
   • grep -c backup /etc/fstab is 0, or the line has nofail and x-systemd.device-timeout=5.
   • systemctl --failed --no-legend | wc -l prints 0.
   • The line for /srv/spool (created by A8's setup) is still present and valid.
   Hint

   journalctl -xb | grep -i mount names the unit. Emergency mode mounts / read-only: mount -o remount,rw / first. Then systemctl daemon-reload and systemctl default.

   Solution

   journalctl -xb --no-pager | grep -iE 'backup|Dependency failed' | head
   mount -o remount,rw /
   findmnt --verify
   sed -i '\#^UUID=0000.* /backup #d' /etc/fstab
   systemctl daemon-reload
   findmnt --verify
   systemctl default

2. PracticeDone

   A2 · Persistent journal and boot arguments (10 points, 8 min)

   On node1:

   1. The journal must survive reboots.
   2. For all installed kernels, remove rhgb and quiet, and add audit=1.
   3. The default boot kernel must be the one currently running.
   4. Write the name of the unit that took longest to start during the current boot into /root/slowest.txt (one line, the unit name only).
   Check your result

   • ls -d /var/log/journal exists, and journalctl --list-boots | wc -l is at least 2 after the final reboot.
   • sudo grubby --info=ALL | grep ^args contains audit=1 and no rhgb or quiet on every entry.
   • sudo grubby --default-kernel equals /boot/vmlinuz-$(uname -r).
   • cat /root/slowest.txt equals the first unit of systemd-analyze blame for that boot.
   Hint

   mkdir -p /var/log/journal && systemctl restart systemd-journald. grubby --update-kernel=ALL --args=... --remove-args=.... The blame list is sorted, longest first.

   Solution

   sudo mkdir -p /var/log/journal && sudo systemd-tmpfiles --create --prefix /var/log/journal
   sudo systemctl restart systemd-journald
   sudo grubby --update-kernel=ALL --args="audit=1" --remove-args="rhgb quiet"
   sudo grubby --set-default /boot/vmlinuz-$(uname -r)
   systemd-analyze blame --no-pager | head -1 | awk '{print $NF}' | sudo tee /root/slowest.txt
   sudo grubby --info=ALL | grep -E '^(args|index)'

3. PracticeDone

   A3 · Capture-ready kdump (10 points, 10 min)

   On node1, configure the machine so that a kernel crash is captured:

   1. Install what is needed and make sure memory is reserved for the capture kernel (use the default for this machine's memory unless you can justify a number).
   2. Dumps are written to /var/crash, using makedumpfile with zero pages, cache pages and free pages excluded (-d 31) and compression.
   3. The service starts at boot and reports operational after the reboot.
   Check your result

   • rpm -q kexec-tools is installed; grep -o 'crashkernel=[^ ]*' /proc/cmdline is non-empty after the final reboot.
   • grep -E '^path /var/crash' /etc/kdump.conf and grep -E '^core_collector .*-d 31' /etc/kdump.conf.
   • systemctl is-enabled kdump is enabled; sudo kdumpctl status prints Kdump is operational.
   Hint

   kdumpctl reset-crashkernel --kernel=ALL sets the default reservation; it applies at the next boot. core_collector makedumpfile -l --message-level 7 -d 31 is the usual line. Restart kdump after editing the file.

   Solution

   sudo dnf install -y kexec-tools
   grep -o 'crashkernel=[^ ]*' /proc/cmdline || sudo kdumpctl reset-crashkernel --kernel=ALL
   sudo sed -i -E 's|^#?path .*|path /var/crash|; s|^#?core_collector .*|core_collector makedumpfile -l --message-level 7 -d 31|' /etc/kdump.conf
   sudo systemctl enable --now kdump
   sudo systemctl reboot
   # after the reboot
   sudo kdumpctl status

4. PracticeDone

   A4 · Web console, metrics and a support archive (12 points, 10 min)

   On node1:

   1. Install the web console and make it available at boot. It must be reachable on port 9090 from other hosts through the firewall (permanently).
   2. Metrics history must be recorded and survive reboots (the console's history page must work).
   3. Create a support archive for case EX342-A4, containing only the networking, systemd and selinux plugin data, written to /root/cases, without being asked any question. Keep the checksum file beside it.
   Check your result

   • systemctl is-enabled cockpit.socket is enabled; sudo firewall-cmd --permanent --list-services contains cockpit.
   • systemctl is-enabled pmlogger is enabled and systemctl is-active pmlogger is active; rpm -q cockpit-pcp.
   • ls /root/cases/sosreport-EX342-A4.tar.xz and the matching .sha256 file exist.
   • tar tf of the archive shows sos_commands/networking/, sos_commands/systemd/ and sos_commands/selinux/ but no sos_commands/kernel/ entries.
   Hint

   systemctl enable --now cockpit.socket; firewall-cmd --permanent --add-service=cockpit then --reload. sos report --batch --case-id EX342-A4 -o networking,systemd,selinux --tmp-dir /root/cases.

   Solution

   sudo dnf install -y cockpit cockpit-pcp sos
   sudo systemctl enable --now cockpit.socket pmlogger
   sudo firewall-cmd --permanent --add-service=cockpit && sudo firewall-cmd --reload
   sudo mkdir -p /root/cases
   sudo sos report --batch --case-id EX342-A4 -o networking,systemd,selinux --tmp-dir /root/cases
   ls /root/cases
   tar tf /root/cases/sosreport-*EX342-A4*.tar.xz | grep -c 'sos_commands/kernel' || true

5. PracticeDone

   A5 · Reset the root password of node2 (12 points, 10 min)

   Nobody knows the root password of node2. Using only its console:

   1. Set the root password of node2 to Ex342!Reset without logging in first.
   2. After the change the system must still be fully functional: SELinux enforcing, correct labels on the authentication files, and the machine booted to its default target.
   Check your result

   • The hash of root on node2 differs from the one before the task, and matches the password: ssh alex@node2.lab.example "sudo python3 -c \"import crypt,spwd; h=spwd.getspnam('root').sp_pwdp; print(crypt.crypt('Ex342!Reset', h)==h)\"" prints True (or log in on the console as root with the new password).
   • ssh alex@node2.lab.example getenforce prints Enforcing; ssh alex@node2.lab.example sudo ls -Z /etc/shadow shows shadow_t.
   • ssh alex@node2.lab.example ls /.autorelabel reports no such file (the relabel has run).
   • ssh alex@node2.lab.example systemctl get-default is multi-user.target (or the graphical target the lab uses) and is-system-running is running.
   Hint

   Edit the kernel line in GRUB and add rd.break. At the prompt: mount -o remount,rw /sysroot, chroot /sysroot, passwd root, touch /.autorelabel, exit, exit. The relabel takes a minute or two.

   Solution

   GRUB: e -> add rd.break to the "linux" line -> Ctrl+x

   mount -o remount,rw /sysroot
   chroot /sysroot
   passwd root          # Ex342!Reset
   touch /.autorelabel
   exit
   exit

6. PracticeDone

   A6 · Repair a damaged XFS and a damaged ext4 (16 points, 12 min)

   On node1, partitions /dev/vdb1 (XFS) and /dev/vdb2 (ext4) refuse to mount and blkid shows nothing for them. Each holds a file ledger.txt that must not be lost.

   1. Repair both file systems. Do not re-create (mkfs) either of them. For ext4 use a backup superblock; find its location without writing to the device.
   2. Mount them permanently, by UUID, on /srv/ledger-xfs and /srv/ledger-ext4.
   3. Write the superblock number you used for ext4 into /root/sb.txt.
   Check your result

   • cat /srv/ledger-xfs/ledger.txt prints ledger-xfs-1; cat /srv/ledger-ext4/ledger.txt prints ledger-ext4-1.
   • findmnt -no FSTYPE,SOURCE /srv/ledger-xfs /srv/ledger-ext4 shows xfs and ext4; grep -E 'ledger-(xfs|ext4)' /etc/fstab uses UUID=.
   • sudo e2fsck -fn /dev/vdb2 ends without errors and xfs_repair -n /dev/vdb1 ends clean (unmount first for the check).
   • cat /root/sb.txt contains 32768 (or another valid backup number printed by mke2fs -n -b 4096 /dev/vdb2).
   • findmnt --verify shows no [E] lines.
   Hint

   xfs_repair -n first, then xfs_repair. For ext4: mke2fs -n -b 4096 /dev/vdb2 lists backups (with -n it only prints), then e2fsck -fy -b NUMBER /dev/vdb2. Run e2fsck -fn again to confirm.

   Solution

   sudo blkid /dev/vdb1 /dev/vdb2; echo "rc=$?"
   sudo xfs_repair -n /dev/vdb1 | head -8
   sudo xfs_repair /dev/vdb1
   sudo mke2fs -n -b 4096 /dev/vdb2 | tail -3
   sudo e2fsck -fy -b 32768 /dev/vdb2 | tail -4
   echo 32768 | sudo tee /root/sb.txt
   sudo e2fsck -fn /dev/vdb2 | tail -1
   sudo mkdir -p /srv/ledger-xfs /srv/ledger-ext4
   printf 'UUID=%s /srv/ledger-xfs xfs defaults 0 0\nUUID=%s /srv/ledger-ext4 ext4 defaults 0 2\n' \
     "$(sudo blkid -s UUID -o value /dev/vdb1)" "$(sudo blkid -s UUID -o value /dev/vdb2)" | sudo tee -a /etc/fstab
   sudo systemctl daemon-reload && sudo mount -a
   cat /srv/ledger-xfs/ledger.txt /srv/ledger-ext4/ledger.txt

7. PracticeDone

   A7 · Restore a lost LVM label and metadata (16 points, 12 min)

   The volume group vgdata (on /dev/vdc) has vanished: pvs and vgs show nothing for that disk. The logical volume lvdata held an XFS file system with a file ledger.txt.

   1. Restore the physical volume and the volume group using the metadata backup LVM keeps. Do not create a new volume group.
   2. Activate the logical volume and mount it permanently on /srv/ledger-lvm.
   3. Keep the original UUIDs: the file system must contain the original ledger.txt.
   Check your result

   • sudo vgs vgdata shows 1 PV and 1 LV; sudo lvs vgdata/lvdata shows the a (active) attribute.
   • cat /srv/ledger-lvm/ledger.txt prints ledger-lvm-1.
   • findmnt /srv/ledger-lvm shows source /dev/mapper/vgdata-lvdata; grep ledger-lvm /etc/fstab has the entry.
   • sudo pvs -o pv_uuid /dev/vdc equals the UUID in the pv0 { id = ... } block of /etc/lvm/backup/vgdata.
   Hint

   The PV UUID is in the pv0 block of /etc/lvm/backup/vgdata, not the first id = (that is the VG). pvcreate -ff --uuid PVUUID --restorefile /etc/lvm/backup/vgdata /dev/vdc, then vgcfgrestore vgdata, then vgchange -ay vgdata. If LVM still ignores the disk, check lvmdevices.

   Solution

   sudo pvs; sudo vgs                                   # nothing for vdc
   sudo vgcfgrestore --list vgdata
   PVU=$(sudo awk '/pv0 \{/{f=1} f&&/id = /{gsub(/"/,"",$3); print $3; exit}' /etc/lvm/backup/vgdata)
   sudo pvcreate -ff -y --uuid "$PVU" --restorefile /etc/lvm/backup/vgdata /dev/vdc
   sudo vgcfgrestore vgdata
   sudo lvmdevices --adddev /dev/vdc 2>/dev/null || true
   sudo vgchange -ay vgdata
   sudo mkdir -p /srv/ledger-lvm
   echo '/dev/mapper/vgdata-lvdata /srv/ledger-lvm xfs defaults 0 0' | sudo tee -a /etc/fstab
   sudo systemctl daemon-reload && sudo mount /srv/ledger-lvm
   cat /srv/ledger-lvm/ledger.txt

8. ChallengeDone

   A8 · Space that cannot be used (12 points, 8 min)

   Two space problems on node1:

   1. Nothing can be created in /srv/spool (No space left on device) although df -h /srv/spool shows plenty free. Fix it by removing only the stale files (stale-.tmp); the keep-.dat files must stay. Then add a daily job (cron or systemd timer, your choice) that deletes stale-*.tmp files older than one day from that directory.
   2. df -h /var shows about 300 MB used by something that du -sh /var cannot find. Release that space without rebooting and without restoring a file; the cause must not return at the next restart of the offending service.
   Check your result

   • touch /srv/spool/newfile succeeds; ls /srv/spool/keep-*.dat | wc -l is unchanged (20); df -i /srv/spool shows IUse% below 90 %.
   • touch -d '3 days ago' /srv/spool/stale-old.tmp; running the job by hand (the cron script or systemctl start CLEANUP.service) removes it but not /srv/spool/stale-new.tmp created now.
   • sudo lsof +L1 2>/dev/null | grep -c deleted is 0; systemctl is-active exporter is active.
   Hint

   Two different kinds of "full": df -i for inodes, and open-but-deleted files (ls -l /proc//fd | grep deleted, lsof +L1). Restarting the owner closes the descriptor and frees the space. A cleanup job needs a command like find /srv/spool -name 'stale-.tmp' -mtime +0 -delete.

   Solution

   df -h /srv/spool | tail -1; df -i /srv/spool | tail -1
   sudo find /srv/spool -name 'stale-*.tmp' -delete
   touch /srv/spool/newfile && ls /srv/spool/keep-*.dat | wc -l
   sudo tee /etc/cron.daily/spool-clean <<'EOT'
   #!/bin/sh
   find /srv/spool -name 'stale-*.tmp' -mtime +0 -delete
   EOT
   sudo chmod 755 /etc/cron.daily/spool-clean
   df -h /var | tail -1; sudo du -sh /var 2>/dev/null | tail -1
   sudo lsof +L1 2>/dev/null | head -3
   sudo systemctl restart exporter
   df -h /var | tail -1