Seeing what a program does with strace and ltrace

The idea

A program that fails with a vague message (“cannot start”, “permission denied”, “error 2”) has usually told the kernel exactly what it was trying to do, and what happened, a moment before. Programs talk to the operating system through system calls (openat, read, connect, execve, bind…). strace shows those calls with their arguments and results. It turns “it fails” into “it fails to open /etc/myapp/app.conf with ENOENT”.

ltrace does the same for calls into shared libraries (malloc, strlen, printf, getaddrinfo), which sit between the program and the kernel.

program  --(library calls: ltrace)-->  shared libraries (glibc, libssl, ...)
                                              |
                                  (system calls: strace)
                                              v
                                            kernel

Reading one line of strace

openat(AT_FDCWD, "/etc/myapp/app.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

Part	Meaning
openat	The system call
AT_FDCWD, "/etc/...", O_RDONLY|O_CLOEXEC	The arguments: directory reference, path, flags
= -1	The return value; -1 means failure for most calls (a file descriptor number such as = 3 means success for open)
ENOENT (No such file or directory)	The errno: why it failed

Errors worth recognising immediately:

errno	Meaning	Look at
ENOENT	No such file or directory	Path, working directory, typos, a missing mount
EACCES	Permission denied	Mode, owner, ACLs, SELinux, ProtectSystem= in a unit
EPERM	Operation not permitted	Needs root or a capability
EEXIST	Already exists	Stale lock or pid file
EADDRINUSE	Address already in use	Another process owns the port
ECONNREFUSED	Nobody listening	Server down, wrong port, firewall REJECT
ETIMEDOUT, EHOSTUNREACH	No answer, no route	Firewall DROP, routing, remote host
EMFILE	Too many open files (this process)	ulimit -n, LimitNOFILE=
ENOSPC	No space left	df -h, df -i

Options you will use

Option	Use
-f	Follow forked children and threads (essential for shell scripts and servers)
-e trace=openat,stat	Show only those calls; classes: file, network, process, memory, signal
-o FILE	Write the trace to a file (stderr stays free for the program)
-p PID	Attach to a running process (needs root, or the same user with ptrace allowed)
-c	Summary: time, calls and errors per call
-tt, -T	Timestamps, and time spent in each call (find where a program waits)
-s 200	Show longer strings (default is truncated to 32 characters)
-y	Show paths for file descriptors

Try it

All outputs are real, from strace and ltrace on a test machine.

A missing file

strace -f -e trace=openat,open,stat -o st.txt ./app.py
grep myapp st.txt

FileNotFoundError: [Errno 2] No such file or directory: '/etc/myapp/app.conf'
309   openat(AT_FDCWD, "/etc/myapp/app.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

The Python traceback already named the file; the value of strace is for programs that do not. Notice the number at the front: it is the process id (shown because of -f).

Where is the time going? A summary

strace -c -f ./app.py 2>&1 | head -6

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 36.34    0.002933          17       171        30 stat

(The top line of the table; the full output lists every call.) A large number in the errors column is suspicious: the program is probing many paths that do not exist. For slowness, -T and -tt tell you which individual call waited.

A refused connection

strace -f -e trace=connect ./net.sh 2>&1 | grep -E 'connect\(3|error'

connect(3, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection refused)
error: [Errno 111] Connection refused

The trace shows the address and the port in numeric form and names the cause. If it were a timeout, the line would end ETIMEDOUT, or the call would simply take long (-T).

Library calls with ltrace

For a small C program that allocates a string and prints it:

ltrace ./hello

malloc(32)                                       = 0x555e430162a0
strlen("labapp")                                 = 6
printf("name=%s len=%zu\n", "labapp", 6)         = 18
free(0x555e430162a0)                             = <void>
name=labapp len=6
+++ exited (status 0) +++

ltrace shows the library functions with arguments, such as which DNS name getaddrinfo was asked to resolve or which file fopen was asked to open. ltrace -S adds system calls to the same output. It works on dynamically linked programs only; a static binary or most Go programs have no library calls to see.

Libraries that will not load

If a program fails with error while loading shared libraries: libfoo.so.1: cannot open shared object file, ldd lists what it needs and which ones are missing:

ldd /usr/local/bin/myapp | grep "not found"
rpm -qf /usr/lib64/libfoo.so.1 || dnf provides '*/libfoo.so.1'

Install the package that provides the library. If the library is in a non-standard directory, add it with a file in /etc/ld.so.conf.d/ and run ldconfig, or set LD_LIBRARY_PATH for a test (never as the permanent fix).

Attaching to a running process

sudo strace -f -p $(pidof httpd | cut -d' ' -f1) -e trace=network -o /tmp/httpd.trace
# press Ctrl+C to stop; the process continues normally

Attaching is safe, but slows the process while tracing; avoid it on a server doing heavy work except briefly. Some hardened systems restrict ptrace (/proc/sys/kernel/yama/ptrace_scope, SELinux deny_ptrace), in which case the attach fails with Operation not permitted even for root.

Going further

Reading outside the program. The common conclusion of a trace is not in the program but in its environment: the file is missing, the permission or label is wrong, the service is not listening. The next lesson covers finding what is open and listening; the one after shows why the environment under systemd differs from your shell. Beyond the exam: perf trace is a lower-overhead alternative to strace on busy systems, ltrace and strace can alter timing enough to hide a race condition, and bpftrace/BCC tools give system-wide views of the same events.