Why?
Degradation of micro-SD cards can be slowed if write access is reduced.
Furthermore, unexpected power cycles are less dangerous if partitions are mounted read-only.
minimum
/etc/fstab
Add ro to options of desired file systems (e.g. /boot).
kernel parameters
Add ro in /boot/cmdline.txt or /boot/grub/grub.cfg, etc. to the kernel command line.
nice-to-haves
pam_tally
- Problem: Without further steps, one cannot log in from the ttys (while remote login via ssh still works).
- Solution: Disable (comment out)
pam_tally.so in /etc/pam.d/system-login or use below fix for log directory (recommended, as otherwise remounting / as rw is not easily reversible).
/var/log
Mount tmpfs on /var/log (who needs logs anyways?), put
none /var/log tmpfs defaults 0 0
into /etc/fstab.
Optionally save old logs on shutdown, put
[Unit]
Description=save /var/log on shutdown
[Service]
Type=oneshot
RemainAfterExit=true
ExecStop=/usr/bin/save-var-log
[Install]
WantedBy=multi-user.target
into /etc/systemd/system/save-var-log.service and
1
2
3
4
5
6
7
8
9
10
11
12
13 | #!/bin/bash
if [ "$(whoami)" != 'root' ]; then
>&2 echo 'must be run as root'
exit 1
fi
cd /var/log || exit $?
tar -czf - * \
| ssh target-host '
cat > "logs/'"$(hostname)"'-var-log-$(date --iso-8601=seconds).tgz"
'
|
into /usr/bin/save-var-log.
Enable by running systemctl enable --now save-var-log.service.
/var/tmp
Problem:
ntpdate.service: Failed to run 'start' task: Read-only file systemntpd.service: Failed to run 'start' task: Read-only file systemhaveged.service: Failed to run 'start-pre' task: Read-only file systemsystemd-resolved.service: Failed to run 'start' task: Read-only file system
Solution: Mount a tmpfs also on /var/tmp.
logrotate
- Problem:
logrotate.service: Failed to run 'start' task: Read-only file system
- Solution: Disable
logrotate.timer, we don't need it (seems to be non-sufficient).
archbuild
- Problem:
archbuild, pacman (and more) need a writable /var/cache
- Solution: Put it on a separate partition (mounted
rw).
- However, because
resize2fs: On-line shrinking not supported, one needs to do the repartitioning offline.
nginx
- Problem:
nginx expects /var/lib/nginx to be writable.
- Solution: Nothing permanent seems to be stored there, so we opt for a tmpfs and put
none /var/lib/nginx tmpfs defaults 0 0
into /etc/fstab.
- Problem:
nginx also expects /var/log/nginx to be existent (and writable).
- Solution: Generate this directory on startup of the unit by putting
[Service]
ExecStartPre=/usr/bin/mkdir -p /var/log/nginx
into /etc/systemd/system/nginx.service.d/override.conf.
troubleshooting
remount after update fails
- Problem:
mount -o remount,ro / fails with mount: /: mount point is busy.
- Cause: Some executables use libraries which were deleted during update.
The used inodes must be freed before remounting.
- Solution: Find offending processes with
Restart offending processes
systemctl status $pid
systemctl restart $found_daemon
- Problem: Sometimes, this does not report any problems, but the remount still fails.
- Solution: It seems,
lsof +L1 does not show files which were deleted and now exist with different content.
Check in /var/log/pacman.log, which packages were updated, and check with pacman -Ql package, which files they own.
Then use lsof | grep /usr/lib/libofupdatedpackage.so to find pids which use those files (most probably only libraries are relevant) and restart them the syme way as above.
- One-command-does-it-all-solution:
lsof / | grep -F <(pacman -Qql $(sed 's@^\['"$(date +%F)"' \S\+ \[ALPM] upgraded \(\S\+\) .*$@\1@;t;d' /var/log/pacman.log) | grep '[^/]') | awk '{print $2}' | while read -r pid; do if [ ${pid} -eq 1 ]; then systemctl daemon-reexec; continue; fi; systemctl status ${pid} 2>/dev/null | head -n1 | grep -v '^Failed to get' | awk '{print $2}'; done | sort -u | xargs -r systemctl restart; mount -o remount,ro /