Sending logs to a central host with rsyslog

Do these on your own system. They get harder. Check each result, then tick it off.

1. Warm-upDone

   Where does this message land?

   Send one test message each with logger -p authpriv.warning -t labtest "auth test" and logger -p mail.info -t labtest "mail test". Find each one in the journal and in the text file that rsyslog writes for it. Name the rule in /etc/rsyslog.conf that decided the file.

   Check your result

   • journalctl -t labtest shows both lines.
   • sudo grep 'auth test' /var/log/secure finds the first; sudo grep 'mail test' /var/log/maillog finds the second.
   • You can quote the lines authpriv.* /var/log/secure and mail.* -/var/log/maillog from /etc/rsyslog.conf.
   Hint

   A facility (authpriv, mail, cron, daemon) plus a priority selects a rule. The rules live in the RULES section of /etc/rsyslog.conf. If /var/log/maillog is missing, rsyslog is not running: systemctl status rsyslog.

   Solution

   systemctl is-active rsyslog
   logger -p authpriv.warning -t labtest "auth test"
   logger -p mail.info -t labtest "mail test"
   journalctl -t labtest --no-pager
   sudo grep 'auth test' /var/log/secure
   sudo grep 'mail test' /var/log/maillog
   grep -E '^(authpriv|mail)' /etc/rsyslog.conf

2. PracticeDone

   Build the log server and the client

   Make node2 a log server and node1 its client:

   1. node2 accepts syslog over TCP port 514 and writes each sender's messages to /var/log/remote/HOSTNAME/PROGRAM.log. The port is open in the firewall permanently and rsyslog is enabled at boot.
   2. node1 forwards every message (all facilities and priorities) to node2 over TCP, with a queue so messages are kept while node2 is down.
   3. Show that a message sent on node1 with logger -t central "from node1" appears on node2.
   Check your result

   • node2: sudo ss -tlnp | grep ':514 ' shows rsyslogd listening; sudo firewall-cmd --list-ports contains 514/tcp; systemctl is-enabled rsyslog is enabled.
   • node1: sudo rsyslogd -N1 ends with End of config validation run; systemctl is-active rsyslog is active.
   • node2: sudo cat /var/log/remote/node1*/central.log contains from node1.
   Hint

   Server: module(load="imtcp"), a template, a ruleset with an omfile action using dynaFile, and an input(type="imtcp" ...) bound to that ruleset. Client: one action(type="omfwd" ...) with protocol="tcp" and a linked-list queue. Validate both files with rsyslogd -N1 before restarting.

   Solution

   # node2 (server)
   sudo tee /etc/rsyslog.d/remote.conf <<'EOT'
   module(load="imtcp")
   template(name="PerHostFile" type="string" string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")
   ruleset(name="remote") {
       action(type="omfile" dynaFile="PerHostFile" dirCreateMode="0750" fileCreateMode="0640")
   }
   input(type="imtcp" port="514" ruleset="remote")
   EOT
   sudo rsyslogd -N1 && sudo systemctl enable --now rsyslog && sudo systemctl restart rsyslog
   sudo firewall-cmd --permanent --add-port=514/tcp && sudo firewall-cmd --reload
   
   # node1 (client)
   sudo tee /etc/rsyslog.d/forward.conf <<'EOT'
   *.* action(type="omfwd" target="node2.lab.example" port="514" protocol="tcp"
              action.resumeRetryCount="-1"
              queue.type="linkedList" queue.size="10000" queue.filename="fwd_node2" queue.saveOnShutdown="on")
   EOT
   sudo rsyslogd -N1 && sudo systemctl restart rsyslog
   logger -t central "from node1"
   
   # node2: prove it
   sudo cat /var/log/remote/node1*/central.log

3. ChallengeDone

   Why do no messages arrive?

   Setup (starting from a working server and client, break three things):

   # on node1
   sudo sed -i 's/port="514" protocol="tcp"/port="5140" protocol="tcp"/' /etc/rsyslog.d/forward.conf
   # on node2
   sudo firewall-cmd --permanent --remove-port=514/tcp && sudo firewall-cmd --reload
   sudo sed -i 's#^template(name="PerHostFile".*#template(name="PerHostFile" type="string" string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")#' /etc/rsyslog.d/remote.conf
   sudo chmod 000 /var/log/remote

   Messages sent with logger -t central "test 2" on node1 no longer arrive. Find and fix every fault, one at a time, with evidence for each (client log, ss, tcpdump or nc, firewall list, server log). Finish with a successful test 2 on node2 and make sure the fix survives a reboot.

   Check your result

   • node2: sudo cat /var/log/remote/node1*/central.log contains test 2.
   • sudo firewall-cmd --permanent --list-ports contains 514/tcp; ls -ld /var/log/remote shows mode drwxr-x--- or similar and owner root.
   • grep -c 5140 /etc/rsyslog.d/forward.conf is 0.
   Hint

   Work from the client outward: does nc -zv node2.lab.example 514 connect? Is the client's own rsyslog log (journal, journalctl -u rsyslog) complaining about the connection? Then look at the server: listening port, firewall, directory permissions and SELinux (ausearch -m avc -c rsyslogd).

   Solution

   # node1: client evidence
   journalctl -u rsyslog -b --no-pager | tail -5          # "cannot connect to ...:5140"
   sudo sed -i 's/port="5140"/port="514"/' /etc/rsyslog.d/forward.conf
   sudo rsyslogd -N1 && sudo systemctl restart rsyslog
   nc -zv node2.lab.example 514                           # still: no route / refused
   # node2: server evidence
   sudo ss -tlnp | grep ':514 '
   sudo firewall-cmd --permanent --add-port=514/tcp && sudo firewall-cmd --reload
   sudo chmod 750 /var/log/remote && sudo restorecon -Rv /var/log/remote
   sudo systemctl restart rsyslog
   # node1
   logger -t central "test 2"
   # node2
   sudo cat /var/log/remote/node1*/central.log