Emergency mode, bad mounts and systemd start-up hangs

The idea

When systemd cannot reach its default target, it does not give up silently. It builds a transaction (the set of units to start and the order), runs them, and for any unit marked as required by the target, a failure either stops the target from being reached or drops the machine into emergency mode. Most start-up hangs and emergency-mode entries trace to a small number of causes.

Cause	Typical sign	Where to look
A bad /etc/fstab line	“You are in emergency mode”, Dependency failed for ... mount	journalctl -xb, findmnt --verify
A required device that never appears	90 second wait, Timed out waiting for device	systemd-analyze blame / critical-chain
An ordering cycle	Transaction order is cyclic; units dropped	systemd-analyze verify
A masked or missing dependency	Unit xyz.service is masked or not found	systemctl list-unit-files, systemctl status
A unit with the wrong Type= or too long a timeout	Start job runs for minutes then fails	systemctl list-jobs during boot, journalctl -u
A failing Requires= unit	The dependent unit “failed because of unavailable resources or another unit”	systemctl status of the requiring unit

Emergency mode, step by step

In emergency mode:

1. Log in with the root password (or use Ctrl+D to try to continue).
2. The root file system is read-only. Before you edit anything, mount -o remount,rw /.
3. Find the failure: journalctl -xb (end with G, search with /), systemctl --failed, findmnt --verify.
4. Fix the cause (usually /etc/fstab).
5. systemctl daemon-reload (systemd generates mount units from /etc/fstab and caches them), then systemctl default to continue to the default target, or reboot for a clean test.

Do not just exit: if the cause is still there you return to the same prompt.

/etc/fstab that does not stop the boot

An /etc/fstab line has six fields; the options field can carry systemd extensions:

Option	Effect
nofail	A failed mount does not make boot fail, and the system does not wait for it
x-systemd.device-timeout=5s	Wait at most 5 seconds for the device (default 90 s)
x-systemd.automount	Mount on first access; the boot never waits for it
_netdev	A network file system: mount after the network is up
noauto	Do not mount at boot at all

Use nofail for removable or optional storage and _netdev for NFS and iSCSI. Never use nofail for a file system the system cannot run without (/, /usr, /var on many systems).

Test every edit:

sudo systemctl daemon-reload
findmnt --verify
sudo mount -a          # mounts anything in fstab not yet mounted; errors appear now, not at boot

Units that depend on each other

Two directives do different jobs, and confusing them causes most unit problems:

• Ordering: After=x / Before=x says when, not whether.
• Requirement: Requires=x (hard) / Wants=x (soft) says whether x must be started too.

Requires= without After= starts both together; After= without Requires= only orders them if both happen to be started. If A is After=B and B is After=A, systemd cannot order them: a cycle.

Try it

The cycle, as systemd reports it

Real output from systemd-analyze verify on two units that each wait for the other:

systemd-analyze verify ./a.service ./b.service

a.service: Found ordering cycle on b.service/start
a.service: Found dependency on a.service/start
a.service: Unable to break cycle starting with a.service/start
Requested transaction contains an unfixable cyclic ordering dependency: Transaction order is cyclic. See system logs for details.
b.service: Command /usr/bin/nothere is not executable: No such file or directory

One command found two separate problems: the cycle and a missing executable. Make systemd-analyze verify UNITFILE a reflex whenever you write or edit a unit, because it reads the unit and its dependencies without starting anything.

What systemd is doing right now

If the boot appears stuck, switch to another console (Ctrl+Alt+F2), log in, and run:

systemctl list-jobs

JOB UNIT                          TYPE  STATE
 94 dev-disk-by\x2dlabel-BACKUP.device start running
 93 backup.mount                  start waiting
 62 graphical.target              start waiting

A job in state running for a long time is what everything else is waiting on. systemctl status of it, and the journal, say why.

Masked and missing dependencies

systemctl list-unit-files | grep masked
systemctl status network-online.target

network-online.target                 masked
Loaded: masked (Reason: Unit network-online.target is masked.)

A masked unit is linked to /dev/null and cannot start, even as a dependency. systemctl unmask UNIT removes the link. A different failure, Unit foo.service not found, means the unit file does not exist (typo in Requires= or a package was removed).

Reading the dependency tree

systemctl list-dependencies app-a.service
systemctl cat app-a.service

list-dependencies shows what a unit pulls in (--reverse shows what pulls it in), and systemctl cat shows the file plus any drop-ins, which is where a surprising After= or an override often hides.

When the boot still hangs

Symptom	Check	Fix
Stops after “Reached target Local File Systems”	list-jobs, journal for network mounts	_netdev, nofail, or a dependency on network-online.target
90 s then emergency mode	Timed out waiting for device	Wrong UUID/label; fix or add nofail
A start job is running for ...	Name in the message	status and journalctl -u of that unit; fix Type= or the timeout
Unit starts then dies in 5 s, repeating	Restart= loop with StartLimit	systemctl status shows start-limit-hit; fix the cause, then systemctl reset-failed
Network-dependent units fail only at boot	They start before the network is up	After=network-online.target and Wants=network-online.target
Works after systemctl start, fails at boot	Ordering/dependency missing, or the unit is not enabled	Add After=; systemctl enable

Going further

The safety net. systemd.unit=rescue.target on the kernel line gets a repair shell without editing the file system first; systemd.mask=unit skips one unit for one boot, which is a quick way to get past a unit that hangs the machine so you can fix it from the inside. Beyond the exam: systemd-analyze plot > boot.svg draws the whole start-up as a timeline, useful for explaining a slow boot to someone else.