Part B: Software and applications (100 points)

Do these on your own system. They get harder. Check each result, then tick it off.

1. Warm-upDone

   B1 · Repair a damaged package (12 points, 8 min)

   On node1, tree and chrony were damaged. Use the RPM database as your evidence:

   1. tree does not run for ordinary users, and its manual page is gone. Restore both without reinstalling anything else.
   2. chronyd was changed so that it has no time sources. Restore /etc/chrony.conf to the packaged default, and restart the service. Do not simply re-type a configuration from memory.
   3. After your repairs, rpm -V tree chrony must report no differences in content or mode.
   Check your result

   • sudo -u nobody /usr/bin/tree --version prints a version; ls -l /usr/bin/tree is -rwxr-xr-x; man -w tree finds the page.
   • grep -c '^pool\|^server' /etc/chrony.conf is at least 1; systemctl is-active chronyd is active.
   • rpm -V tree chrony prints nothing (a .......T. timestamp-only line on a documentation file is tolerated).
   Hint

   rpm -V tree shows .M....... and missing d. dnf reinstall tree restores both. For chrony the reinstall keeps your modified config: dnf download chrony, rpm2cpio ... | cpio -idm ./etc/chrony.conf, then cp -p the extracted file over the live one.

   Solution

   rpm -V tree chrony
   sudo dnf reinstall -y tree
   cd /tmp && dnf download chrony
   rpm2cpio chrony-*.rpm | cpio -idm ./etc/chrony.conf
   sudo cp -p /tmp/etc/chrony.conf /etc/chrony.conf
   sudo restorecon -v /etc/chrony.conf
   sudo systemctl restart chronyd
   rpm -V tree chrony; echo "rc=$?"

2. PracticeDone

   B2 · A repository that does not work (10 points, 7 min)

   The packages labtools and labdocs are in the directory /srv/labrepo and should be installable from a repository with the id lab defined in /etc/yum.repos.d/lab.repo. They are unsigned test packages.

   1. Make dnf repolist show the repository lab with its packages. Find and fix every reason it does not (there is more than one).
   2. Install labdocs from it. Do not touch the other repositories.
   Check your result

   • ls /srv/labrepo/repodata/repomd.xml exists.
   • dnf repolist | grep -w lab shows the repository with a package count of 3 or more.
   • rpm -q labdocs is installed; rpm -qi labdocs | grep 'Install Date'.
   • grep -E '^(baseurl|gpgcheck)' /etc/yum.repos.d/lab.repo shows a baseurl that points at an existing directory, and gpgcheck=0 (the only repository where this is acceptable here: the packages are unsigned test files).
   Hint

   dnf makecache shows the first failure. Check the baseurl path against ls /srv, then whether repodata exists (createrepo_c), then what the signature check says about unsigned packages.

   Solution

   sudo dnf makecache 2>&1 | tail -3
   grep baseurl /etc/yum.repos.d/lab.repo; ls /srv
   sudo sed -i 's#^baseurl=.*#baseurl=file:///srv/labrepo#' /etc/yum.repos.d/lab.repo
   sudo dnf install -y createrepo_c
   sudo createrepo_c /srv/labrepo
   sudo sed -i 's/^gpgcheck=.*/gpgcheck=0/' /etc/yum.repos.d/lab.repo
   sudo dnf clean all && dnf repolist
   sudo dnf install -y labdocs

3. PracticeDone

   B3 · Signatures, keys and rolling back (12 points, 8 min)

   On node1:

   1. /root/pkgs/labsigned-1.0-1.noarch.rpm is signed, and its public key is /root/pkgs/RPM-GPG-KEY-lab. Check the package before importing the key and save the one-word result for the signature line (NOKEY or OK) into /root/sig-before.txt. Then import the key and install the package. The check must say digests signatures OK afterwards.
   2. An unplanned update installed labtools 2.0, which breaks a script. Return to version 1.0 using the repository, and make sure dnf does not offer 2.x again.
   Check your result

   • cat /root/sig-before.txt is NOKEY; rpm -qa 'gpg-pubkey*' --qf '%{SUMMARY}\n' | grep -c 'Lab Packager' is 1.
   • rpm -K /root/pkgs/labsigned-1.0-1.noarch.rpm prints digests signatures OK; rpm -q labsigned is installed.
   • rpm -q labtools prints labtools-1.0-1.noarch.
   • dnf check-update labtools; echo $? prints 0 (no update offered).
   Hint

   rpm -Kv names the signature result per line. rpm --import FILE brings a key in. dnf downgrade labtools-1.0 needs the repository from B2 working. Stop 2.x with exclude=labtools-2* in the repository file, or the versionlock plugin.

   Solution

   rpm -Kv /root/pkgs/labsigned-1.0-1.noarch.rpm | grep -i signature | awk '{print $NF}' | sudo tee /root/sig-before.txt
   sudo rpm --import /root/pkgs/RPM-GPG-KEY-lab
   rpm -K /root/pkgs/labsigned-1.0-1.noarch.rpm
   sudo dnf install -y /root/pkgs/labsigned-1.0-1.noarch.rpm
   sudo dnf downgrade -y labtools-1.0
   echo 'exclude=labtools-2*' | sudo tee -a /etc/yum.repos.d/lab.repo
   rpm -q labtools; dnf check-update labtools; echo "rc=$?"

4. PracticeDone

   B4 · Analyse a crash (14 points, 12 min)

   ledgerd.service (the program /usr/local/bin/ledgerd) crashes at start. You cannot change the program.

   1. Using the recorded crash dump, write /root/crash.txt with exactly three lines: FUNCTION= (the function of ledgerd's own code where the crash happened), LINE= (its source line number) and VARIABLE= (the pointer that was NULL, written as in the source, e.g. c->name).
   2. Make the service start and stay running by correcting its configuration, not the program.
   Check your result

   • cat /root/crash.txt is FUNCTION=name_length, LINE=23, VARIABLE=c->name (the expression name is accepted for the variable).
   • coredumpctl list ledgerd --no-pager shows at least one SIGSEGV entry.
   • systemctl is-active ledgerd is active and systemctl is-enabled ledgerd is enabled.
   • grep -c '^name=' /etc/ledgerd.conf is at least 1.
   Hint

   coredumpctl debug ledgerd opens gdb; bt shows frames, the first one from ledgerd's own file is the one you need (frame 0 is inside the C library). frame 1, print *c. The program reads /etc/ledgerd.conf and looks for name= and interval= lines.

   Solution

   coredumpctl list ledgerd --no-pager
   coredumpctl debug ledgerd -A "-batch -ex bt -ex 'frame 1' -ex 'print *c'"
   printf 'FUNCTION=name_length\nLINE=23\nVARIABLE=c->name\n' | sudo tee /root/crash.txt
   echo 'name=ledger' | sudo tee -a /etc/ledgerd.conf
   sudo systemctl restart ledgerd; systemctl is-active ledgerd

5. ChallengeDone

   B5 · A service that will not start (16 points, 12 min)

   invoiced.service does not start and stays failed. It must run as the user invoice, at boot, with the following facts about the application: it needs the variable INVOICE_MODE (value prod), it reads /etc/invoiced/api.key, and it writes its state to /var/lib/invoiced/state. There is more than one problem; each only appears once the previous is fixed.

   Fix the service without changing the program or relaxing the security settings in the unit that are not the cause, and without making the key readable by everyone.

   Check your result

   • systemctl is-active invoiced is active; systemctl is-enabled invoiced is enabled.
   • sudo cat /var/lib/invoiced/state contains started.
   • ls -l /etc/invoiced/api.key shows group invoice and mode 640 (or stricter than 644, with no "other" access).
   • systemctl show invoiced -p ProtectSystem still prints ProtectSystem=strict, and the unit lists ReadWritePaths=/var/lib/invoiced.
   • systemctl show invoiced -p EnvironmentFiles names a file that exists and grep INVOICE_MODE in it prints prod.
   Hint

   Read systemctl status invoiced, then journalctl -u invoiced -b, then systemctl cat invoiced. Typical exit statuses: 200/CHDIR (a directory), a message about the environment file, status 5 (the key), status 6 (writing the state). Fix them in this order, restarting after each.

   Solution

   systemctl status invoiced --no-pager | sed -n 1,12p
   systemctl cat invoiced
   sudo mkdir -p /opt/invoiced                                   # WorkingDirectory
   echo 'INVOICE_MODE=prod' | sudo tee /etc/sysconfig/invoiced  # EnvironmentFile
   sudo systemctl restart invoiced; journalctl -u invoiced -n 3 --no-pager
   sudo chgrp invoice /etc/invoiced/api.key && sudo chmod 640 /etc/invoiced/api.key
   sudo chown invoice: /var/lib/invoiced
   sudo mkdir -p /etc/systemd/system/invoiced.service.d
   printf '[Service]\nReadWritePaths=/var/lib/invoiced\n' | sudo tee /etc/systemd/system/invoiced.service.d/rw.conf
   sudo systemctl daemon-reload && sudo systemctl restart invoiced
   systemctl is-active invoiced; sudo cat /var/lib/invoiced/state

6. PracticeDone

   B6 · Limits and a port conflict (12 points, 8 min)

   Two services are failing on node1:

   1. fdhog.service dies with a "Too many open files" error. Raise its limit to 1024 in the way that survives a reboot and does not edit the packaged/original unit file.
   2. stats.service cannot bind TCP port 9100. Find out which process holds the port and why it runs. That process belongs to the retired legacy-stats.service, which must not run any more (not now, not after a reboot). Then start stats.service and keep it enabled.
   Check your result

   • systemctl show fdhog -p LimitNOFILE prints LimitNOFILE=1024; systemctl is-active fdhog is active.
   • ls /etc/systemd/system/fdhog.service.d/*.conf exists; the original /etc/systemd/system/fdhog.service still has LimitNOFILE=64.
   • systemctl is-enabled legacy-stats is disabled or masked and systemctl is-active legacy-stats is not active.
   • sudo ss -tlnp 'sport = :9100' shows the stats program; systemctl is-active stats is active and systemctl is-enabled stats is enabled.
   Hint

   journalctl -u fdhog has Errno 24. systemctl edit fdhog creates a drop-in. ss -tlnp 'sport = :9100' names the PID; systemctl status PID (or ps -o unit= -p PID) names its unit.

   Solution

   journalctl -u fdhog -n 3 --no-pager
   sudo mkdir -p /etc/systemd/system/fdhog.service.d
   printf '[Service]\nLimitNOFILE=1024\n' | sudo tee /etc/systemd/system/fdhog.service.d/limits.conf
   sudo systemctl daemon-reload && sudo systemctl restart fdhog
   
   sudo ss -tlnp 'sport = :9100'
   sudo systemctl status $(sudo ss -tlnpH 'sport = :9100' | grep -o 'pid=[0-9]*' | cut -d= -f2) --no-pager | head -3
   sudo systemctl disable --now legacy-stats
   sudo systemctl enable --now stats
   systemctl is-active fdhog stats

7. PracticeDone

   B7 · Find what a program is looking for (8 points, 6 min)

   /usr/local/bin/nightly exits silently with a non-zero status, and its source code is not available. Without guessing:

   1. Find out which file it tries to open, and which TCP address and port it connects to afterwards. Write them to /root/nightly.txt as FILE=/path and PORT=number.
   2. Provide what it needs so that nightly exits with status 0, in a way that survives a reboot (the file must exist and something must listen on that port at boot).
   Check your result

   • cat /root/nightly.txt is FILE=/var/lib/nightly/ready.flag and PORT=7777.
   • nightly; echo $? prints nightly: ok and 0.
   • systemctl is-enabled nightly-sink is enabled (or any enabled unit that listens on 127.0.0.1:7777: ss -tln | grep 7777).
   Hint

   strace -f -e trace=openat,connect nightly 2>&1 | grep -E 'ENOENT|connect'. The first failing call decides the exit status; fix it, run again, and the next failure appears. A throwaway listener as a service: python3 -m http.server 7777 --bind 127.0.0.1 in a unit.

   Solution

   nightly; echo "rc=$?"
   strace -e trace=openat,connect nightly 2>&1 | grep -E 'ready|connect'
   printf 'FILE=/var/lib/nightly/ready.flag\nPORT=7777\n' | sudo tee /root/nightly.txt
   sudo mkdir -p /var/lib/nightly && sudo touch /var/lib/nightly/ready.flag
   sudo tee /etc/systemd/system/nightly-sink.service <<'EOT'
   [Unit]
   Description=Listener for nightly
   [Service]
   ExecStart=/usr/bin/python3 -m http.server 7777 --bind 127.0.0.1
   [Install]
   WantedBy=multi-user.target
   EOT
   sudo systemctl daemon-reload && sudo systemctl enable --now nightly-sink
   nightly; echo "rc=$?"

8. ChallengeDone

   B8 · One playbook for both nodes (16 points, 12 min)

   In /home/alex/ansible on node1 there is an inventory (group lab with both nodes), an ansible.cfg and a playbook site.yml that was written in a hurry. As user alex:

   1. Install the Ansible command-line tools on node1 (nothing on node2 except Python).
   2. Make site.yml pass --syntax-check and run successfully against both nodes. It must: install and enable chrony, create /etc/lab-release containing lab: ex342 and a line host: followed by the host's inventory name, and set PermitRootLogin no in /etc/ssh/sshd_config.d/01-local.conf, reloading sshd only when that setting changed.
   3. The run must be idempotent: a second run reports changed=0 for both nodes.
   Check your result

   • rpm -q ansible-core on node1; ansible-playbook -i inventory site.yml --syntax-check prints only the playbook name.
   • First run: ends with failed=0 for both nodes. Second run: both nodes show changed=0 ... failed=0.
   • ssh node2 cat /etc/lab-release prints lab: ex342 and host: node2; ssh node2 sudo sshd -T | grep -i '^permitrootlogin' is permitrootlogin no.
   • systemctl is-enabled chronyd is enabled on both nodes.
   Hint

   Run ansible-playbook --syntax-check first: it finds a missing colon and (with a run) a handler that is never called because the names differ in case. Writing under /etc needs become: true. Read the bottom of an error first.

   Solution

   sudo dnf install -y ansible-core
   cd /home/alex/ansible
   ansible-playbook -i inventory site.yml --syntax-check
   # fix: the colon after ansible.builtin.service, become: true on the play,
   # and notify: Reload sshd (same spelling as the handler name)
   sed -i 's/ansible.builtin.service$/ansible.builtin.service:/; s/notify: reload sshd/notify: Reload sshd/' site.yml
   sed -i 's/^  hosts: lab$/  hosts: lab\n  become: true/' site.yml
   ansible-playbook -i inventory site.yml
   ansible-playbook -i inventory site.yml | tail -4