Firewalls, SELinux ports and proving it with tcpdump

The idea

When a connection fails and the routes and names are right, three more things can still be in the way on the server: nothing is listening, a firewall blocks it, or SELinux stops the service from using the port at all. They produce different symptoms, and a packet capture lets you see which one you have without guessing.

Reading a TCP connection in a capture

TCP opens with three packets, the handshake. tcpdump prints each with its flags:

Flag text	Packet	Direction
[S]	SYN: “I want to connect”	client to server
[S.]	SYN-ACK: “OK, me too”	server to client
[.]	ACK: “connected”	client to server
[P.]	data (PSH+ACK)	either
[F.]	finish: close	either
[R.]	reset: “no” or “abort”	either

What the capture on the server shows tells you the layer:

Capture on the server	Meaning	Client sees
[S] then [S.]	Listener present and firewall open	Connects
[S] then [R.]	Host answered “closed”: nothing listens (or a REJECT rule with tcp-reset)	Connection refused, immediately
[S], [S], [S]… with growing gaps and no reply	Packet arrived and was silently discarded, e.g. firewall DROP	Connection timed out
Nothing arrives	Lost before the server: routing, a firewall on the path, a wrong address	Timeout
ICMP port unreachable / admin prohibited reply	REJECT (firewalld’s default for blocked traffic)	Connection refused or No route to host

Where you capture matters: on the server you see what reached it; on the client you see what it sent and what came back.

tcpdump in practice

sudo tcpdump -nn -i any 'tcp port 8080'               # one port
sudo tcpdump -nn -i eth0 'host 192.168.50.11 and tcp port 22'
sudo tcpdump -nn -i any 'icmp'                        # pings and ICMP errors
sudo tcpdump -nn -i any -w /tmp/web.pcap 'port 80'    # write for Wireshark; -r to read it back
sudo tcpdump -nn -i any -c 10 'tcp[tcpflags] & tcp-syn != 0'   # only connection attempts

-nn avoids DNS and service-name lookups (which are slow and mask what you see). -c N stops after N packets. A capture needs root, and never leave -w running unattended: files grow quickly.

firewalld on top of nftables

On RHEL 10 firewalld programs nftables. You manage it with firewall-cmd; nft list ruleset shows the result. Two states exist for every setting:

firewall-cmd --get-active-zones            # which zones are in use and which interfaces/sources belong to them
firewall-cmd --list-all                    # the default zone, with services, ports, rich rules
firewall-cmd --zone=internal --list-all    # a specific zone
firewall-cmd --get-default-zone
firewall-cmd --add-port=8080/tcp           # runtime only: lost at reload/reboot
firewall-cmd --permanent --add-port=8080/tcp   # stored, active after --reload
firewall-cmd --reload                      # apply permanent rules, drop runtime-only changes

The usual mistake is a rule added without --permanent (works now, gone after reboot) or with --permanent and no --reload (stored, not yet active). Compare firewall-cmd --list-ports with firewall-cmd --permanent --list-ports.

The zone that applies to a packet depends on the interface it arrived on or its source address. If your rule is in public and the interface is in internal, the rule never matches. Find the zone first: firewall-cmd --get-zone-of-interface=eth0.

SELinux and ports

SELinux controls which port numbers a confined service may use. Apache (httpd_t) may bind to ports labelled http_port_t (80, 81, 443, 8008, 8009, 8443, 9000, among others). To use another port, label it:

sudo semanage port -l | grep -w http_port_t
sudo semanage port -a -t http_port_t -p tcp 8500

An unlabelled port gives a service that fails to start with Permission denied/(13)Permission denied: AH00072: make_sock: could not bind to address, and an AVC with { name_bind } in ausearch -m avc. The firewall is not involved: the service never began to listen.

Try it

A refused connection, in packets

Real output: tcpdump on the loopback while nc connected to a closed port 9.

sudo tcpdump -nn -i lo -l 'tcp port 9' &
nc -zv 127.0.0.1 9

nc: connect to 127.0.0.1 port 9 (tcp) failed: Connection refused
17:30:10.899588 IP 127.0.0.1.51574 > 127.0.0.1.9: Flags [S], seq 188140946, win 65495, options [mss 65495,sackOK,TS val 142171407 ecr 0,nop,wscale 10], length 0
17:30:10.899602 IP 127.0.0.1.9 > 127.0.0.1.51574: Flags [R.], seq 0, ack 188140947, win 0, length 0

[S] from client port 51574 to port 9, and 14 microseconds later [R.]: the kernel of the destination replied “reset”. That is refused: you reached the host; the port is closed.

The same connection, dropped

An nftables rule that drops TCP port 9 was added (a real ruleset):

nft add table inet labt
nft add chain inet labt input '{ type filter hook input priority 0; policy accept; }'
nft add rule inet labt input tcp dport 9 drop
nft list ruleset

table inet labt {
	chain input {
		type filter hook input priority filter; policy accept;
		tcp dport 9 drop
	}
}

Now the client:

time nc -zv -w 3 127.0.0.1 9

nc: connect to 127.0.0.1 port 9 (tcp) timed out: Operation now in progress

real	0m3.005s

and the capture:

17:30:16.955637 IP 127.0.0.1.51586 > 127.0.0.1.9: Flags [S], seq 395230208, ...
17:30:17.972522 IP 127.0.0.1.51586 > 127.0.0.1.9: Flags [S], seq 395230208, ...
17:30:18.996507 IP 127.0.0.1.51586 > 127.0.0.1.9: Flags [S], seq 395230208, ...

The same seq number on every line means the client is retransmitting the same SYN, about a second apart, and never gets an answer. That is a drop: the packet arrived and was discarded. A firewalld zone with the default policy reject sends an ICMP error instead, so the client would give up at once with Connection refused or No route to host rather than waiting.

(The test table was removed afterwards with nft delete table inet labt.)

On RHEL: where is the rule?

sudo firewall-cmd --get-active-zones
sudo firewall-cmd --list-all
sudo nft list ruleset | grep -n 8500

public
  interfaces: eth0
public (active)
  target: default
  services: cockpit dhcpv6-client ssh
  ports: 8500/tcp
  rich rules:

(Representative output.) If 8500/tcp appears in --list-all but not in --permanent --list-ports, the rule is runtime-only. If the interface is in a different zone than the one you edited, edit the zone it is in.

The service view

sudo ss -tlnp | grep -E '8500|httpd'
sudo ausearch -m avc -ts recent | tail -2
journalctl -u httpd -b --no-pager | tail -3

LISTEN 0 511 127.0.0.1:8500 0.0.0.0:* users:(("httpd",pid=7021,fd=4))
type=AVC msg=audit(1730721120.114:521): avc:  denied  { name_bind } for  pid=7015 comm="httpd" src=8500 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=0
httpd[7015]: (13)Permission denied: AH00072: make_sock: could not bind to address [::]:8500

(Representative output.) Read the three together: a listener on 127.0.0.1 only (explains the remote Connection refused), a name_bind denial for port 8500 from httpd_t (SELinux; fix with the port label), and Apache’s own message of the failed bind.

The order to follow

1. Is it listening, and on what? ss -tlnp on the server. Not listening: the service, its configuration, or SELinux (journalctl -u, ausearch -m avc).
2. Is it listening on the right address? 127.0.0.1 is not reachable from outside.
3. Does it answer locally? curl from the server. If not, it is the application.
4. Does the packet arrive? tcpdump on the server while the client connects.
5. Is the firewall open for that zone, permanently? firewall-cmd --list-all and --permanent --list-ports, for the zone the interface is in.
6. Is anything in between? Routers, cloud security groups, other firewalls: capture on both ends and compare.

Going further

Logging dropped packets. firewall-cmd --set-log-denied=all logs rejected packets to the kernel log (watch with journalctl -k -f); it is noisy, so switch it back to off after use. Beyond the exam: direct nftables rules should be avoided on a firewalld system, as the two can conflict; use rich rules or firewall-cmd --direct knowingly. ss -s and nstat -az | grep -i retrans show whether a path is silently losing packets.