Mounting and fstab¶
Mounting attaches a filesystem to a directory in the single Linux tree, and /etc/fstab makes the attachment permanent. A wrong fstab line is one of the most common reasons a server stops in emergency mode after a reboot.
Track: Core · Interview weight: High
Must-Know Facts¶
| Fact | Value | Verify with |
|---|---|---|
| Mount and unmount | mount <dev> <dir>, umount <dir> (not "unmount") | findmnt <dir> |
| Show mounts | findmnt (tree), findmnt -t xfs, mount, /proc/self/mountinfo | findmnt -no OPTIONS / |
| fstab fields | device, mount point, type, options, dump, fsck pass | man 5 fstab |
| Device field | UUID=, LABEL=, PARTUUID= or /dev/...; UUID survives renames | blkid |
| fsck pass | 1 root, 2 other local filesystems, 0 skip (always 0 for XFS) | cat /etc/fstab |
| Test fstab | findmnt --verify, then mount -a | exit status |
| systemd | systemd-fstab-generator turns each line into a .mount unit; daemon-reload after edits | systemctl list-units -t mount |
nofail | Boot continues if the device is missing | systemctl show -p WantedBy <unit> |
_netdev | Wait for the network (NFS, iSCSI); implied for network filesystem types | man systemd.mount |
x-systemd.device-timeout= | How long boot waits for the device (default 90 s) | systemctl cat <unit> |
| Common options | ro, noatime, noexec, nosuid, nodev, defaults (rw,suid,dev,exec,auto,nouser,async) | findmnt -no OPTIONS <dir> |
| Busy target | fuser -vm <dir>, lsof +f -- <dir>; umount -l detaches lazily | umount <dir> |
| vfat and NTFS ownership | Set at mount time with uid=, gid=, umask= | ls -l |
Mounting and Mount Options¶
The volumes from Filesystems are mounted at /srv/app (ext4) and /srv/logs (XFS):
findmnt /srv/logs
findmnt -t ext4,xfs -o TARGET,SOURCE,FSTYPE,OPTIONS
sudo mount -o remount,ro /srv/app
sudo touch /srv/app/test; echo "rc=$?"
findmnt -no OPTIONS /srv/app
sudo mount -o remount,rw /srv/app
Output:
TARGET SOURCE FSTYPE OPTIONS
/srv/logs /dev/loop1p2 xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
TARGET SOURCE FSTYPE OPTIONS
/ /dev/vda ext4 rw,relatime,stripe=4
├─/srv/app /dev/loop0p3 ext4 rw,relatime
└─/srv/logs /dev/loop1p2 xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
touch: cannot touch '/srv/app/test': Read-only file system
rc=1
ro,relatime
The kernel adds its own defaults (relatime, the XFS log options), so findmnt shows more than was requested. relatime updates access times only when they are older than the modification time; noatime skips them entirely.
Writing fstab Entries¶
sudo umount /srv/app /srv/logs
sudo blkid -s UUID -o value /dev/loop0p3 /dev/loop1p2
sudo cp /etc/fstab /etc/fstab.bak
sudo tee -a /etc/fstab <<'EOF'
UUID=683f9429-6087-4267-a91e-79c8e5787e21 /srv/app ext4 defaults,noatime 0 2
LABEL=weblogs /srv/logs xfs defaults,nofail 0 0
EOF
sudo findmnt --verify
sudo mount -a
Output:
683f9429-6087-4267-a91e-79c8e5787e21
875008e0-8369-4a85-b33f-06333db4f59d
UUID=683f9429-6087-4267-a91e-79c8e5787e21 /srv/app ext4 defaults,noatime 0 2
LABEL=weblogs /srv/logs xfs defaults,nofail 0 0
0 parse errors, 0 errors, 1 warning
[W] your fstab has been modified, but systemd still uses the old version;
use 'systemctl daemon-reload' to reload
mount -a mounts every auto entry that is not mounted yet and prints nothing on success. After a reload, the check is clean and systemd shows the generated units:
findmnt -t ext4,xfs -o TARGET,SOURCE,FSTYPE,OPTIONS
sudo systemctl daemon-reload
sudo findmnt --verify; echo "rc=$?"
systemctl cat srv-app.mount
systemctl show -p RequiredBy,WantedBy srv-app.mount srv-logs.mount
Output:
TARGET SOURCE FSTYPE OPTIONS
/ /dev/vda ext4 rw,relatime,stripe=4
├─/srv/app /dev/loop0p3 ext4 rw,noatime
└─/srv/logs /dev/loop1p2 xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Success, no errors or warnings detected
rc=0
# /run/systemd/generator/srv-app.mount
# Automatically generated by systemd-fstab-generator
[Unit]
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
SourcePath=/etc/fstab
Before=local-fs.target
Requires=systemd-fsck@dev-disk-by\x2duuid-683f9429\x2d6087\x2d4267\x2da91e\x2d79c8e5787e21.service
# ... (trimmed)
[Mount]
What=/dev/disk/by-uuid/683f9429-6087-4267-a91e-79c8e5787e21
Where=/srv/app
Type=ext4
Options=defaults,noatime
RequiredBy=local-fs.target
WantedBy=
RequiredBy=
WantedBy=local-fs.target
Pass 2 added a systemd-fsck@ dependency. local-fs.target requires /srv/app but only wants /srv/logs, which is what nofail changes: a missing /srv/logs device no longer fails the target that the rest of the boot waits for.
Run findmnt --verify before every reboot after an fstab edit
It catches unknown devices, missing mount points and bad types without mounting anything. mount -a then proves the entries actually mount.
A Broken Entry¶
A typo in the UUID, with the device timeout shortened to 10 seconds so the boot-time behavior can be reproduced on a running system:
echo 'UUID=683f9429-0000-4267-a91e-79c8e5787e21 /srv/data xfs defaults,x-systemd.device-timeout=10s 0 0' | sudo tee -a /etc/fstab
sudo findmnt --verify; echo "rc=$?"
sudo mount -a; echo "rc=$?"
sudo systemctl daemon-reload
sudo systemctl start srv-data.mount; echo "rc=$?"
journalctl -b -o cat --no-pager | grep -E 'srv-data|Timed out waiting' | grep -v COMMAND | tail -3
Output:
UUID=683f9429-0000-4267-a91e-79c8e5787e21 /srv/data xfs defaults,x-systemd.device-timeout=10s 0 0
0 parse errors, 2 errors, 1 warning
/srv/data
[E] unreachable on boot required target: No such file or directory
[E] unreachable on boot required source: UUID=683f9429-0000-4267-a91e-79c8e5787e21
[W] your fstab has been modified, but systemd still uses the old version;
use 'systemctl daemon-reload' to reload
rc=1
mount: /srv/data: can't find UUID=683f9429-0000-4267-a91e-79c8e5787e21.
rc=32
A dependency job for srv-data.mount failed. See 'journalctl -xe' for details.
rc=1
Timed out waiting for device dev-disk-by\x2duuid-683f9429\x2d0000\x2d4267\x2da91e\x2d79c8e5787e21.device - /dev/disk/by-uuid/683f9429-0000-4267-a91e-79c8e5787e21.
Dependency failed for srv-data.mount - /srv/data.
srv-data.mount: Job srv-data.mount/start failed with result 'dependency'.
At boot, the same failure makes local-fs.target fail after the default 90 seconds, and systemd drops to emergency mode (You are in emergency mode). The fix there is to log in as root, run mount -o remount,rw /, correct or comment out the line, and reboot. The line was removed here with sudo sed -i '/srv\/data/d' /etc/fstab and a daemon-reload.
One bad fstab line can stop a cloud instance from booting
Cloud VMs often have no console login, so emergency mode means detaching the root volume and fixing it from another instance. Add nofail to every data volume and test with findmnt --verify.
Target Is Busy¶
A process with its working directory or an open file on the filesystem blocks the unmount:
sudo mkdir -p /srv/app/data && sudo chown laborant: /srv/app/data
(cd /srv/app/data && sleep 300) &
tail -f /dev/null > /srv/app/data/app.log &
sleep 1
sudo umount /srv/app; echo "rc=$?"
sudo fuser -vm /srv/app
sudo lsof +f -- /srv/app
kill %1 %2; wait
sudo umount /srv/app; echo "rc=$?"
sudo mount /srv/app
Output:
umount: /srv/app: target is busy.
rc=32
USER PID ACCESS COMMAND
/srv/app: root kernel mount /srv/app
laborant 4573 ..c.. sleep
laborant 4574 F.... tail
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sleep 4573 laborant cwd DIR 259,0 1024 12 /srv/app/data
tail 4574 laborant 1w REG 259,0 0 13 /srv/app/data/app.log
rc=0
In the fuser access column, c is a current directory and F a file open for writing. umount -l detaches the mount point at once and finishes when the last user closes it, which hides the problem rather than solving it; fuser -km kills every user of the filesystem.
Security Options and Special Mounts¶
printf '#!/bin/bash\necho hello from $0\n' > /srv/app/data/hello.sh; chmod +x /srv/app/data/hello.sh
sudo mount -o remount,noexec /srv/app
/srv/app/data/hello.sh; echo "rc=$?"
bash /srv/app/data/hello.sh
sudo mount -o remount,exec /srv/app
sudo mkdir -p /mnt/efi /var/www
sudo mount --bind /srv/app/data /var/www
findmnt /var/www
sudo mount -t tmpfs -o size=64M,mode=1777 tmpfs /mnt
df -h /mnt | tail -1
sudo umount /mnt /var/www
Output:
bash: line 4: /srv/app/data/hello.sh: Permission denied
rc=126
hello from /srv/app/data/hello.sh
TARGET SOURCE FSTYPE OPTIONS
/var/www /dev/loop0p3[/data] ext4 rw,noatime
tmpfs 64M 0 64M 0% /mnt
noexec blocks direct execution, but an interpreter still reads the script, so it is a hardening layer for /tmp, /dev/shm and upload directories, not a security boundary. The bind mount shows the subdirectory in brackets. A tmpfs lives in RAM and swap and is empty after every mount.
FAT filesystems store no owners or modes, so the mount options decide them:
sudo mount /dev/loop1p1 /mnt/efi
sudo touch /mnt/efi/a; ls -l /mnt/efi
sudo umount /mnt/efi
sudo mount -o uid=laborant,gid=laborant,umask=077 /dev/loop1p1 /mnt/efi
ls -l /mnt/efi
sudo chmod 644 /mnt/efi/a; echo "rc=$?"; ls -l /mnt/efi
sudo umount /mnt/efi
Output:
total 0
-rwxr-xr-x 1 root root 0 Sep 17 07:36 a
total 0
-rwx------ 1 laborant laborant 0 Sep 17 07:36 a
rc=0
total 0
-rwx------ 1 laborant laborant 0 Sep 17 07:36 a
chmod returned success and changed nothing. USB sticks and the EFI partition behave the same way.
Native Mount Units¶
A .mount unit replaces an fstab line and must be named after its mount point (systemd-escape -p --suffix=mount /srv/cache prints the name). A copy named cache.mount was installed next to the correct one:
sudo tee /etc/systemd/system/srv-cache.mount <<'EOF'
[Unit]
Description=Scratch space for the app cache
[Mount]
What=tmpfs
Where=/srv/cache
Type=tmpfs
Options=size=64M,mode=1777
[Install]
WantedBy=local-fs.target
EOF
sudo cp /etc/systemd/system/srv-cache.mount /etc/systemd/system/cache.mount
sudo systemctl daemon-reload
sudo systemctl start cache.mount; echo "rc=$?"
sudo systemctl enable --now srv-cache.mount
findmnt /srv/cache
journalctl -b -o cat --no-pager -u cache.mount
Output:
# ... (trimmed)
Failed to start cache.mount: Unit cache.mount has a bad unit file setting.
See system logs and 'systemctl status cache.mount' for details.
rc=1
Created symlink '/etc/systemd/system/local-fs.target.wants/srv-cache.mount' → '/etc/systemd/system/srv-cache.mount'.
TARGET SOURCE FSTYPE OPTIONS
/srv/cache tmpfs tmpfs rw,relatime,size=65536k
cache.mount: Where= setting doesn't match unit name. Refusing.
A matching .automount unit (or x-systemd.automount in fstab) mounts the filesystem on first access, which suits network shares.
Common Errors¶
umount: /srv/app: target is busy.¶
Cause: a process has its working directory, an open file or a memory mapping on the filesystem, or another filesystem is mounted below it.
Fix: sudo fuser -vm <dir> or sudo lsof +f -- <dir>, stop those processes, and check findmnt -R <dir> for nested mounts.
mount: /srv/data: can't find UUID=683f9429-0000-4267-a91e-79c8e5787e21.¶
Cause: no block device carries that UUID: a typo, a recreated filesystem with a new UUID, or a disk that is not attached.
Fix: compare with sudo blkid; correct the line, or add nofail for removable and optional devices.
Interview Checkpoints¶
L1: What are the six fields of an /etc/fstab line?
Say first: device, mount point, filesystem type, options, dump flag and fsck pass number.
Proof: grep -v '^#' /etc/fstab; man 5 fstab.
Follow-up: Why is the pass number 0 for XFS? (XFS checks itself at mount time; fsck.xfs does nothing.)
L1: Why use UUIDs in fstab instead of /dev/sdb1?
Say first: device names follow detection order and can change after a reboot or disk change, while the UUID belongs to the filesystem.
Proof: sudo blkid; ls -l /dev/disk/by-uuid.
Follow-up: When does the UUID change?
L1: What does nofail do?
Say first: the mount becomes wanted instead of required by local-fs.target, so boot continues when the device is missing.
Proof: systemctl show -p RequiredBy,WantedBy <unit>.mount.
Follow-up: Which option waits for the network before mounting? (_netdev.)
L2: Mount a new XFS volume permanently at /data and prove it survives a reboot.
Say first: add a UUID line to fstab, verify it and mount it with mount -a.
Proof: echo "UUID=$(sudo blkid -s UUID -o value /dev/sdb1) /data xfs defaults,nofail 0 0" | sudo tee -a /etc/fstab; sudo systemctl daemon-reload; sudo findmnt --verify; sudo mount -a; findmnt /data.
Follow-up: How do you test without rebooting that the unit works? (sudo umount /data; sudo systemctl start data.mount.)
L2: A filesystem cannot be unmounted. Find out why.
Say first: list the processes using it.
Proof: sudo fuser -vm /data; sudo lsof +f -- /data; findmnt -R /data.
Follow-up: What does umount -l do, and why is it risky before removing a disk?
L2: Make /tmp non-executable without a reboot.
Say first: remount it with noexec, and add the option to its fstab line or tmp.mount drop-in.
Proof: sudo mount -o remount,noexec /tmp; findmnt -no OPTIONS /tmp.
Follow-up: Why does bash /tmp/script.sh still work?
L3: After a reboot, the server is in emergency mode. What do you check?
Say first: a mount failed; find which one in the journal, then fix fstab.
Proof: journalctl -xb | grep -iE 'timed out|dependency failed'; systemctl --failed; mount -o remount,rw /; findmnt --verify.
Follow-up: How would you recover a cloud VM without console access?
L3: A filesystem suddenly became read-only on a running server. Why?
Say first: the kernel remounted it read-only after an I/O error or detected corruption (ext4 errors=remount-ro).
Proof: findmnt -no OPTIONS /data; journalctl -k | grep -iE 'ext4-fs error|i/o error|remount'.
Follow-up: Why does mount -o remount,rw not fix it?
Related¶
- Filesystems: creating and repairing the volumes mounted here
- Unit Files: dependencies such as
Requires=andWants= - Disk Usage: full and read-only filesystems
- Swap: swap entries in fstab
Captured on Rocky Linux 10.2 (iximiuz Labs microVM, kernel 6.1.167, util-linux 2.40.2, systemd 257), 2026-09.