Open files, listening ports and resource limits

The idea

A process can fail because of what is around it rather than what it does. It cannot open its file because another program owns it, cannot bind its port because another process already has it, runs out of descriptors, or is killed because the machine ran out of memory. Each of these is visible in the kernel’s records of the process.

Everything is a file descriptor

A Linux process refers to open files, directories, sockets, pipes and devices by small integers called file descriptors. The kernel exposes them under /proc/PID/fd, and lsof (“list open files”) reads and formats that.

ls -l /proc/PID/fd            # descriptors and what they point to
lsof -p PID                   # same, with type, device, size and name
lsof /var/log/messages        # which processes have this file open?
lsof -nP -iTCP -sTCP:LISTEN   # every TCP listener (numeric)
lsof +D /mnt/data             # who is using this mounted directory? (why umount says "busy")
fuser -vm /mnt/data           # shorter form of the same question
ss -tulpn                     # listening sockets with owning processes

Question	Command
Which process listens on port 443?	ss -tlnp 'sport = :443' or lsof -nP -iTCP:443 -sTCP:LISTEN
Why can’t I unmount /mnt/data?	lsof +f -- /mnt/data or fuser -vm /mnt/data
Which process holds a deleted file open?	ls -l /proc/*/fd 2>/dev/null | grep deleted or lsof +L1
What is this process connected to?	lsof -nP -a -p PID -i
Which libraries are loaded?	lsof -p PID | grep mem or cat /proc/PID/maps

Limits

Every process has soft and hard limits for several resources, readable in /proc/PID/limits:

grep -E 'open files|processes|core file|address space' /proc/PID/limits
ulimit -n        # the soft limit on open files in the current shell

Where they come from depends on how the process started:

Started by	Limits come from
An interactive login (ssh, console)	pam_limits and /etc/security/limits.conf, limits.d/*.conf
A systemd service	The unit: LimitNOFILE=, LimitNPROC=, LimitCORE=… (defaults in /etc/systemd/system.conf)
A cron job	pam_limits for the cron session

This is the cause of a very common surprise: you raise nofile in limits.conf, log in, and ulimit -n shows the new value, but the service still fails because it never went through PAM. The fix is a drop-in for the unit.

Memory and the OOM killer

When the system, or a cgroup such as a systemd service with MemoryMax=, cannot satisfy a memory request, the kernel’s OOM killer selects a process and kills it. The evidence is in the kernel log, not in the application’s own log (it was killed and had no chance to write anything):

journalctl -k -b | grep -iE 'out of memory|oom-kill|Killed process'
systemctl status UNIT          # "Result: oom-kill" for a service in its own cgroup
cat /proc/PID/oom_score        # how attractive this process is as a victim (higher = more likely)

The score favours big processes; OOMScoreAdjust=-900 in a unit, or writing -900 to /proc/PID/oom_score_adj, protects an essential process, and MemoryMax= contains a likely offender before it hurts others.

Try it

Who owns a port

Real output: one process listening, then a second one trying to bind the same address and port.

python3 -c "..." &                       # listener on 127.0.0.1:8099
python3 -c "import socket; s=socket.socket(); s.bind(('127.0.0.1',8099))"
lsof -nP -iTCP:8099 -sTCP:LISTEN

OSError: [Errno 98] Address already in use
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
python3  377 root    3u  IPv4   2153      0t0  TCP 127.0.0.1:8099 (LISTEN)

The error is EADDRINUSE, and lsof names the owner. On RHEL, sudo ss -tlnp 'sport = :8099' gives the same answer in one line, and it is installed by default (package iproute). If you need to start a new service on that port, either stop the other process or change the port. For a web service on a non-default port, remember SELinux (semanage port -l) and the firewall are separate from “address in use”.

What a process has open

sleep 100 &
ls /proc/$!/fd | wc -l
lsof -p $! | head -5

3
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
sleep   366 root  cwd    DIR  254,0     4096 1204669 /tmp/ad
sleep   366 root  rtd    DIR  254,0     4096       2 /
sleep   366 root  txt    REG  254,0    35336  321702 /usr/bin/sleep
sleep   366 root  mem    REG  254,0  2129424  322137 /usr/lib/x86_64-linux-gnu/libc.so.6

cwd is the working directory, rtd the root, txt the program, mem a memory-mapped library; numbered descriptors (0u, 1u, 3r) follow. The cwd line explains many “device is busy” errors: a shell sitting in a mount point holds it open.

Hitting a limit

ulimit -n 20
python3 -c "
fs=[]
try:
  for i in range(100): fs.append(open('/etc/hostname'))
except OSError as e: print(len(fs),'open:',e)"

17 open: [Errno 24] Too many open files: '/etc/hostname'

With a limit of 20, three descriptors were already in use (standard input, output, error), so 17 more opened and the 18th failed with EMFILE. In a real service the descriptors are sockets: a server with 1,024 clients and a limit of 1,024 fails the same way. Read the limit of the live process:

grep 'open files' /proc/$(pidof -s fdhog)/limits
systemctl show fdhog -p LimitNOFILE

Raise it with a drop-in:

[Service]
LimitNOFILE=1024

Then systemctl daemon-reload && systemctl restart fdhog. Check /proc/PID/limits again: only a process started after the change has the new limit.

The OOM killer’s trail

journalctl -k -b | grep -iE 'out of memory|oom-kill|Killed process'

kernel: Memory cgroup out of memory: Killed process 4412 (python3) total-vm:211540kB, anon-rss:102308kB, file-rss:2820kB, shmem-rss:0kB, UID:0 pgtables:268kB oom_score_adj:0
systemd[1]: memhog.service: A process of this unit has been killed by the OOM killer.
systemd[1]: memhog.service: Failed with result 'oom-kill'.

Memory cgroup out of memory says the group exceeded its own limit (here MemoryMax=100M), not that the machine was out of memory. A global event reads Out of memory: Killed process ... and is preceded by a Mem-Info block describing free memory and swap. The numbers anon-rss tell you how large the process was when it died, which is the starting point for the question “is this a leak or an undersized limit?”.

Going further

Raising limits in the right place. Use LimitNOFILE= for a service, limits.d/*.conf for interactive users. The system-wide ceiling is fs.file-max and the per-process ceiling fs.nr_open (sysctl fs.nr_open); a limit above nr_open is refused. Beyond the exam: MemoryHigh= throttles a cgroup before MemoryMax= kills it, systemd-oomd can act on memory pressure before the kernel does, and earlyoom is a user-space alternative. The monitoring lesson shows how to see memory pressure and swap before an OOM event happens.