Why?

Degradation of micro-SD cards can be slowed if write access is reduced. Furthermore, unexpected power cycles are less dangerous if partitions are mounted read-only.

minimum

/etc/fstab

Add ro to options of desired file systems (e.g. /boot).

kernel parameters

Add ro in /boot/cmdline.txt or /boot/grub/grub.cfg, etc. to the kernel command line.

nice-to-haves

pam_tally

• Problem: Without further steps, one cannot log in from the ttys (while remote login via ssh still works).
• Solution: Disable (comment out) pam_tally.so in /etc/pam.d/system-login or use below fix for log directory (recommended, as otherwise remounting / as rw is not easily reversible).

/var/log

Mount tmpfs on /var/log (who needs logs anyways?), put

none /var/log tmpfs defaults 0 0

into /etc/fstab. Optionally save old logs on shutdown, put

[Unit]
Description=save /var/log on shutdown

[Service]
Type=oneshot
RemainAfterExit=true
ExecStop=/usr/bin/save-var-log

[Install]
WantedBy=multi-user.target

into /etc/systemd/system/save-var-log.service and

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

#!/bin/bash

if [ "$(whoami)" != 'root' ]; then
  >&2 echo 'must be run as root'
  exit 1
fi

cd /var/log || exit $?

tar -czf - * \
| ssh target-host '
  cat > "logs/'"$(hostname)"'-var-log-$(date --iso-8601=seconds).tgz"
'

into /usr/bin/save-var-log. Enable by running systemctl enable --now save-var-log.service.

/var/tmp

Problem:

• ntpdate.service: Failed to run 'start' task: Read-only file system
• ntpd.service: Failed to run 'start' task: Read-only file system
• haveged.service: Failed to run 'start-pre' task: Read-only file system
• systemd-resolved.service: Failed to run 'start' task: Read-only file system

Solution: Mount a tmpfs also on /var/tmp.

logrotate

• Problem: logrotate.service: Failed to run 'start' task: Read-only file system
• Solution: Disable logrotate.timer, we don't need it (seems to be non-sufficient).

archbuild

• Problem: archbuild, pacman (and more) need a writable /var/cache
• Solution: Put it on a separate partition (mounted rw).
• However, because resize2fs: On-line shrinking not supported, one needs to do the repartitioning offline.

nginx

• Problem: nginx expects /var/lib/nginx to be writable.
• Solution: Nothing permanent seems to be stored there, so we opt for a tmpfs and put

none /var/lib/nginx tmpfs defaults 0 0

into /etc/fstab.

• Problem: nginx also expects /var/log/nginx to be existent (and writable).
• Solution: Generate this directory on startup of the unit by putting

[Service]
ExecStartPre=/usr/bin/mkdir -p /var/log/nginx

into /etc/systemd/system/nginx.service.d/override.conf.

troubleshooting

remount after update fails

• Problem: mount -o remount,ro / fails with mount: /: mount point is busy.
• Cause: Some executables use libraries which were deleted during update. The used inodes must be freed before remounting.
• Solution: Find offending processes with

lsof +L1

Restart offending processes

• for pid 1:

systemctl daemon-reexec

• for other pids:

systemctl status $pid
systemctl restart $found_daemon

• Problem: Sometimes, this does not report any problems, but the remount still fails.
• Solution: It seems, lsof +L1 does not show files which were deleted and now exist with different content. Check in /var/log/pacman.log, which packages were updated, and check with pacman -Ql package, which files they own. Then use lsof | grep /usr/lib/libofupdatedpackage.so to find pids which use those files (most probably only libraries are relevant) and restart them the syme way as above.
• One-command-does-it-all-solution:

lsof / | grep -F <(pacman -Qql $(sed 's@^\['"$(date +%F)"' \S\+ \[ALPM] upgraded \(\S\+\) .*$@\1@;t;d' /var/log/pacman.log) | grep '[^/]') | awk '{print $2}' | while read -r pid; do if [ ${pid} -eq 1 ]; then systemctl daemon-reexec; continue; fi; systemctl status ${pid} 2>/dev/null | head -n1 | grep -v '^Failed to get' | awk '{print $2}'; done | sort -u | xargs -r systemctl restart; mount -o remount,ro /