Addresses, routes and name resolution

The idea

Three kinds of settings put a machine on the network: its address (who am I), its routes (how do I reach others), and its resolvers (how do names become addresses). On RHEL all three are owned by NetworkManager, which stores them in profiles (also called connections) and applies them to devices (interfaces).

device (eth0)  <--- activated with --->  connection profile (settings stored in a keyfile)
   |                                           |
   +-- live state: ip addr, ip route           +-- /etc/NetworkManager/system-connections/NAME.nmconnection

The difference between the two halves is the root of most “it worked until the reboot” problems:

• Commands such as ip addr add, ip route add and echo ... > /etc/resolv.conf change the live state only.
• nmcli connection modify changes the profile; nmcli connection up (or nmcli device reapply) applies it to the device.

On RHEL 10, profiles are keyfiles, plain INI files. The older ifcfg files in /etc/sysconfig/network-scripts/ are no longer used; if you find one on an upgraded machine, it was migrated or is ignored. After editing a keyfile directly, run nmcli connection reload and bring the connection up again.

Reading and changing a profile

nmcli device status                         # devices and their state
nmcli connection show                       # all profiles
nmcli connection show NAME                  # every setting of a profile
nmcli -f ipv4 connection show NAME          # only the ipv4 group
nmcli connection modify NAME ipv4.method manual ipv4.addresses 192.168.50.12/24 ipv4.gateway 192.168.50.1
nmcli connection modify NAME +ipv4.dns 192.168.50.1      # + adds, - removes
nmcli connection up NAME                    # apply

Setting	Meaning
ipv4.method	auto (DHCP), manual (static), disabled, link-local
ipv4.addresses	ADDR/PREFIX, comma-separated for several
ipv4.gateway	The default gateway; sets the default route
ipv4.dns, ipv4.dns-search	DNS servers and search domains
ipv4.ignore-auto-dns	Ignore DNS handed out by DHCP
ipv4.routes	Static routes: "10.20.0.0/16 192.168.50.254 100" (network, next hop, optional metric)
ipv4.never-default	Do not use this connection for the default route
connection.autoconnect	Activate at boot

Routes: which way out?

ip route get ADDRESS answers the real question “which route would be used for this destination?”

ip route
ip route get 10.20.1.5

default via 192.168.50.1 dev eth0 proto static metric 100
10.20.0.0/16 via 192.168.50.254 dev eth0 proto static metric 100
192.168.50.0/24 dev eth0 proto kernel scope link src 192.168.50.12 metric 100
10.20.1.5 via 192.168.50.254 dev eth0 src 192.168.50.12 uid 1000
    cache

(Representative output.) The most specific route (longest prefix) wins; default is used when nothing else matches. ip route get also shows the source address that will be used. A missing default route gives Network is unreachable for anything off-subnet; a default route through a gateway that does not answer gives timeouts. Check the gateway: ip neigh show 192.168.50.1 should show a MAC address with state REACHABLE, STALE or DELAY; FAILED or INCOMPLETE means ARP got no answer.

Names: who answers?

A name lookup in an application goes through the C library, which consults sources in the order given in /etc/nsswitch.conf:

hosts:      files myhostname dns

Step	Source	Notes
files	/etc/hosts	First match wins; an old line here beats DNS
myhostname	The machine’s own host name
dns	The servers in /etc/resolv.conf	Asked in order; the next one is tried only after a timeout

/etc/resolv.conf on RHEL is written by NetworkManager from the active profile (a header says so). Edits are lost at the next reconnect. Change ipv4.dns and ipv4.dns-search in the profile instead.

Tools differ in what they test:

Tool	Uses	Use it for
getent hosts NAME	The full order (nsswitch.conf)	“What will applications get?”
dig NAME / dig NAME @SERVER	DNS only	“What does this DNS server say?”
host NAME, nslookup	DNS only	Quick lookups
ping NAME, curl NAME	The full order	End-to-end

A disagreement between dig and getent always means the answer comes from somewhere other than DNS (/etc/hosts), or from a different DNS server than the one you named with @.

Try it

Static configuration, persistently

CON=$(nmcli -g NAME connection show --active | head -1)
sudo nmcli connection modify "$CON" ipv4.method manual \
  ipv4.addresses 192.168.50.12/24 ipv4.gateway 192.168.50.1 \
  ipv4.dns 192.168.50.1 ipv4.dns-search lab.example \
  +ipv4.routes "10.20.0.0/16 192.168.50.254"
sudo nmcli connection up "$CON"
ip -br addr
cat /etc/resolv.conf

lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UP             192.168.50.12/24 fe80::5054:ff:fe12:3456/64
# Generated by NetworkManager
search lab.example
nameserver 192.168.50.1

The stored profile now has the same values:

sudo cat /etc/NetworkManager/system-connections/eth0.nmconnection

[connection]
id=eth0
type=ethernet
interface-name=eth0

[ipv4]
address1=192.168.50.12/24,192.168.50.1
dns=192.168.50.1;
dns-search=lab.example;
method=manual
route1=10.20.0.0/16,192.168.50.254

Name disagreement

getent hosts app.lab.example
dig +short app.lab.example @192.168.50.1

192.168.50.99   app.lab.example
192.168.50.20

The two answers differ because getent read /etc/hosts first (files is before dns) and dig asked the DNS server. Remove the stale line from /etc/hosts, and getent follows. If instead both dig and getent give the wrong address, the DNS server itself is wrong (stale record), and you must fix it there or point to another server.

Duplicate addresses and ARP

Two hosts with the same IP address cause intermittent failures: some packets reach one, some the other. Check:

sudo arping -D -I eth0 -c2 192.168.50.12    # DAD: is another host answering for my address?
ip neigh show | grep 192.168.50.12

A reply to arping -D means a conflict. The neighbour’s MAC address in ip neigh changing back and forth is the same sign.

When it still does not work

Symptom	Cause	Fix
Setting disappears after reboot	Used ip or edited resolv.conf	Put it in the profile with nmcli connection modify
ipv4.gateway set but no default route	Method is auto with a different route, or never-default=yes	Check ip route, nmcli connection show NAME | grep default
Two default routes	Two active connections	Lower the metric of the preferred one (ipv4.route-metric), or ipv4.never-default yes on the other
Connection activating forever	DHCP unavailable	Check the link, VLAN, DHCP server; use ipv4.method manual
Slow logins, slow sudo	A dead DNS server first in the list; reverse lookups timing out	Fix the order; UseDNS no for sshd is a workaround only
Name resolves to IPv6, connect fails	AAAA record with no IPv6 route	curl -4; fix IPv6 or remove the AAAA record
Profile edited by hand has no effect	Not reloaded	nmcli connection reload && nmcli connection up NAME

Going further

Hostnames. hostnamectl set-hostname node1.lab.example writes /etc/hostname; the static name, transient name and pretty name are separate (hostnamectl status). Services that “cannot find themselves” are usually missing a line in /etc/hosts for their own name. Beyond the exam: bonds, bridges and VLANs are also NetworkManager profiles with a master relationship; nmcli device reapply applies changes without a full reconnect where possible; and systemd-resolved is available but not the default on RHEL.