Works in my shell, fails as a service

The idea

When a program works from your terminal but fails under systemctl start, the program is usually not the problem. The context changed. A terminal gives a program your identity, your environment, your working directory and a login session. A service gets none of that, unless the unit asks for it.

What differs

Aspect	Interactive shell	systemd service
User	You (or root via sudo)	User= in the unit, otherwise root
Environment	Your login environment, PATH, HOME, LANG	Very small: PATH is basic; nothing from ~/.bashrc; add with Environment= / EnvironmentFile=
Working directory	Where you are	/, unless WorkingDirectory=
Input/output	A terminal	No terminal; output goes to the journal
Limits	pam_limits from login	LimitNOFILE= and friends
SELinux domain	unconfined_t for admins	The domain defined by the executable’s label (for example httpd_t)
Filesystem view	The whole system	May be restricted: ProtectSystem=, ProtectHome=, PrivateTmp=, ReadWritePaths=
Network	Any	May be restricted: PrivateNetwork=, IPAddressDeny=
Timing	You start it when ready	Started at boot, perhaps before the network or a mount is ready

An administrator logged in as root is usually unconfined_t: SELinux does not block that user. The same program as a service is confined, which is the most common reason it “works for me as root and fails as a service”.

Reproducing the service context

Do not guess. Run the same command in the same context:

sudo systemd-run --wait --pipe --uid=svcapp /usr/local/bin/svcapp        # as the service user, minimal environment
sudo systemd-run -t --uid=svcapp --property=WorkingDirectory=/ /bin/bash  # an interactive shell inside a service-like context
sudo -u svcapp env -i /usr/local/bin/svcapp                              # simpler: other user, empty environment
systemctl show UNIT -p User,Group,WorkingDirectory,Environment,ExecStart # what the unit really has
systemctl cat UNIT                                                       # the unit plus drop-ins

systemd-run creates a transient unit, so it has the same cgroup and limit handling as a real service, and --property= lets you copy settings from the real unit (--property=ProtectSystem=strict reproduces a sandbox problem).

The four layers

When you have the same failure by hand, work through layers in order of cost:

1. Unit and process: exit status (status=203/EXEC, 3/...), journalctl -u UNIT, systemctl show UNIT. Does the executable exist and is it executable, is User= valid, is the Type= right?
2. File permissions and ownership: sudo -u USER ls -l PATH, namei -l /full/path/to/file (shows permissions of every component of a path: a directory on the way may deny access).
3. SELinux: ls -lZ, ps -eZ | grep PROCESS, ausearch -m avc -ts recent. Never setenforce 0 as a fix.
4. Network: ss -tlnp, firewall-cmd --list-all, semanage port -l for the port label.
5. Sandbox directives in the unit: systemd-analyze security UNIT lists the restrictions and a score; ProtectSystem=strict makes the file system read-only for the service except for ReadWritePaths=.

Try it

The environment, side by side

A tiny script prints who it runs as, its directory, HOME, PATH and a variable.

MYAPP_MODE=dev /usr/local/bin/envdump
sudo systemd-run --wait --pipe /usr/local/bin/envdump

user=alex cwd=/home/alex HOME=/home/alex PATH=/home/alex/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
MYAPP_MODE=dev
Running as unit: run-r3b7e1a0e2b8c4cb9a8c0e2f2d3d4e5f6.service
user=root cwd=/ HOME=/root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
MYAPP_MODE=unset

(Representative output.) Notice the working directory /, no personal PATH entries, and the missing variable. A script that calls a tool in ~/bin, reads a relative config.ini, or expects MYAPP_MODE fails under systemd and works for you. Fix it in the unit, not by relying on a shell: absolute paths, WorkingDirectory=, Environment= or EnvironmentFile=/etc/sysconfig/myapp.

Reproducing as the service user

sudo -u svcapp /usr/local/bin/svcapp; echo "rc=$?"

cat: /etc/svcapp/secret.conf: Permission denied
rc=3

Now the journal’s status=3 makes sense, and the fix is a narrow one: give group read access to the file for the service’s group, not world-read.

sudo chown root:svcapp /etc/svcapp/secret.conf
sudo chmod 640 /etc/svcapp/secret.conf
namei -l /etc/svcapp/secret.conf

f: /etc/svcapp/secret.conf
dr-xr-xr-x root root /
drwxr-xr-x root root etc
drwxr-xr-x root root svcapp
-rw-r----- root svcapp secret.conf

namei -l is a very good command for permission puzzles: it prints the permissions of each directory on the path, because a missing x bit on any of them blocks access to everything below it.

SELinux and sandbox evidence

Permissions fine, still Permission denied? Then ask SELinux and the sandbox:

sudo ausearch -m avc -ts recent | tail -4
systemctl cat webdemo | grep -E 'Protect|ReadWrite|PrivateTmp'
sudo systemd-analyze security webdemo --no-pager | tail -5

type=AVC msg=audit(1730718311.221:412): avc:  denied  { name_bind } for  pid=4455 comm="webdemo" src=8443 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=0
ProtectSystem=strict
→ Overall exposure level for webdemo.service: 4.2 OK 🙂

(Representative output.) The AVC line has the story in one place: who (scontext, domain httpd_t), what (tcontext, the port is unreserved_port_t), and which operation (name_bind on a tcp_socket). Allowing it means labelling the port: semanage port -a -t http_port_t -p tcp 8443. audit2why (or sealert -a /var/log/audit/audit.log) translates AVC lines into plain suggestions, but read them: do not pipe audit2allow into a policy module without understanding what it permits.

When the unit still fails

Symptom	Cause	Fix
status=203/EXEC	Path wrong, no execute bit, bad label, missing interpreter	ls -lZ, file, first line of the script
status=217/USER	User= does not exist	Create it or fix the name
status=200/CHDIR	WorkingDirectory= missing	Create the directory
Starts, then exits immediately with code 0	Type=simple but the program forks and exits	Type=forking with PIDFile=, or run in foreground
Works after a manual start, fails at boot	Starts before network or mount	After=network-online.target, RequiresMountsFor=/data
Read-only file system for a path the user can write	ProtectSystem=, ProtectHome=	ReadWritePaths=
Cannot reach the network	PrivateNetwork=, firewall, SELinux connect denial	Check each; booleans such as httpd_can_network_connect
Cannot write to /tmp shared with another service	PrivateTmp=true	Use a real shared directory
Environment variable missing	Not set in the unit	Environment=, EnvironmentFile=

Going further

Reading the whole unit. systemctl cat and systemctl show are the first stop, and drop-ins in /etc/systemd/system/UNIT.d/ are where surprises hide. systemd-delta lists every unit that overrides a packaged one. Beyond the exam: systemd-analyze security scores a unit’s exposure and suggests directives to tighten, and for confined services, sandbox-style restrictions are cheaper than writing SELinux policy. Learn to read AVC denials fluently before ever writing policy.