Diagnosing a network fault layer by layer

Do these on your own system. They get harder. Check each result, then tick it off.

1. Warm-upDone

   Match symptom to layer

   For each message, name the layer that failed and the next command: (1) Network is unreachable; (2) Could not resolve host: app.lab.example; (3) Connection refused; (4) Connection timed out after a long wait; (5) No route to host; (6) a browser shows 502 Bad Gateway.

   Check your result

   • (1) no route in the table, ip route; (2) name resolution, getent hosts / dig; (3) the host is reachable, nothing listens on that port, ss -tlnp on the server; (4) packets dropped by a firewall or lost on the path, tcpdump and the firewall rules; (5) the local host knows a route but ARP or an ICMP reply says unreachable, check the neighbour and the gateway; (6) the application layer: the proxy cannot reach its backend, read the proxy log.
   Hint

   Refused is an answer (an RST or ICMP came back). Timed out is the absence of an answer. They need different next steps.

   Solution

   Network is unreachable   -> routing table (no matching route)      ip route
   Could not resolve host   -> DNS / hosts / nsswitch                 getent hosts NAME
   Connection refused       -> reached, port closed                   ss -tlnp (server)
   Connection timed out     -> filtered or lost                       tcpdump, firewall
   No route to host         -> ARP failed or ICMP unreachable         ip neigh, gateway
   502 Bad Gateway          -> application layer                      proxy and backend logs

2. PracticeDone

   Walk a connection down the stack

   From node1, the service at http://node2.lab.example:8080/ is unreachable. Without assuming the cause, run the layered checks in order from node1 and record the result of each: link state, own address, default route, the gateway's neighbour entry, name resolution of node2.lab.example, ICMP to its address, TCP to port 8080, and an HTTP request. Stop and report at the first failure with the evidence.

   Check your result

   • Your list has these commands in this order: ip -br link, ip -br addr, ip route, ip neigh, getent hosts node2.lab.example, ping -c2, nc -zv node2.lab.example 8080, curl -v http://node2.lab.example:8080/.
   • You stopped at the first failure and named it with one line of evidence.
   Hint

   Each check depends on the one before it: a failing ping to a name and a working ping to its address means DNS, not the network. Test by address first, then by name.

   Solution

   ip -br link
   ip -br addr
   ip route
   ip neigh show
   getent hosts node2.lab.example
   ping -c2 192.168.50.12
   nc -zv node2.lab.example 8080
   curl -v --max-time 5 http://node2.lab.example:8080/

3. ChallengeDone

   Three faults in one path

   Setup (lab VMs node1 client and node2 server; run on node1 unless stated):

   # node2: serve something on 8080
   python3 -m http.server 8080 --bind 0.0.0.0 &
   # node1: break it (CON is the name of the active profile)
   CON=$(nmcli -g NAME connection show --active | head -1)
   sudo nmcli connection modify "$CON" ipv4.gateway 192.168.50.99
   sudo nmcli connection up "$CON"
   echo "192.168.50.13 node2.lab.example" | sudo tee -a /etc/hosts

   and on node2 sudo firewall-cmd --remove-port=8080/tcp (if open). From node1 curl http://node2.lab.example:8080/ fails. There are three independent faults. Find them one at a time, in layer order, with evidence for each, fix them persistently, and finish with a successful request.

   Check your result

   • curl -s http://node2.lab.example:8080/ | head -3 returns HTML.
   • ip route | grep default shows the real gateway; getent hosts node2.lab.example returns node2's real address; sudo firewall-cmd --permanent --list-ports on node2 contains 8080/tcp.
   • The wrong /etc/hosts line is gone.
   Hint

   Layer order: route (gateway), name (a stale /etc/hosts entry overrides DNS because hosts: files dns in nsswitch.conf reads the file first), then the firewall on the server. Use getent hosts, not dig, to see what applications see.

   Solution

   ip route; ping -c2 192.168.50.99 || echo "gateway unreachable"
   CON=$(nmcli -g NAME connection show --active | head -1)
   sudo nmcli connection modify "$CON" ipv4.gateway 192.168.50.1
   sudo nmcli connection up "$CON"
   getent hosts node2.lab.example; dig +short node2.lab.example
   sudo sed -i '/node2.lab.example/d' /etc/hosts
   nc -zv node2.lab.example 8080
   # node2:
   sudo ss -tlnp | grep 8080
   sudo firewall-cmd --permanent --add-port=8080/tcp && sudo firewall-cmd --reload
   # node1
   curl -s http://node2.lab.example:8080/ | head -3