CRITICAL BUG #2 in storage guard, caught by live test v8 on
sapphira (2026-10-10).
RequiresMountsFor=/home/ooyude/External in the unit file told systemd
that the service depends on the mount. When the mount was gone and
the service was started, systemd's dependency resolver AUTOMATICALLY
REMOUNTED the filesystem to satisfy the dependency — THEN checked
ConditionPathIsMountPoint. The condition saw the just-remounted
filesystem and evaluated to true. Service started on empty
external storage. Guard completely bypassed.
This is a subtle interaction:
- ConditionPathIsMountPoint is a TEST (true/false evaluation)
- RequiresMountsFor is a DEPENDENCY (systemd must make it true)
A guard should be a TEST, not a dependency that makes the test
trivially pass. Remove the dependency. The condition alone is
sufficient for both boot-time and runtime checks:
- Boot: mount unit starts via local-fs.target, condition is true
- Runtime: if mount disappears, condition becomes false on next
start attempt. Without RequiresMountsFor, systemd doesn't
auto-recover, so the guard fires.
Ordering should be expressed via After= in the consumer's
systemd.services block (not in the shared helper), e.g.:
systemd.services.postgresql.after = [ "home-ooyude-External.mount" ];
Live test v8 sequence (before this fix):
1. umount /home/ooyude/External → OK, gone from /proc/mounts
2. findmnt /home/ooyude/External → exit=1, not in table
3. systemctl start postgresql → STARTED (guard bypassed)
4. journal: no ConditionPath error (service started successfully)
This is the second guard bug found by live testing in this session.
The first one was the '!' prefix inversion. Both were invisible to
nix eval, both only visible at runtime. Lesson reinforced:
guards MUST be live-tested, not just statically evaluated.
Recovery: v8 test reverted state via emergency_recovery trap
(remount + restart all services). System healthy at 10/10.
CRITICAL BUG FIX caught by live test v6 on sapphira.
The '!' prefix INVERTS the systemd test:
ConditionPathIsMountPoint=!/path → test passes if path is NOT a mount
→ unit STARTS when storage is unmounted
→ exactly the opposite of what we want
Correct semantics for a storage guard:
ConditionPathIsMountPoint=/path → test passes if path IS a mount
→ unit starts ONLY when storage is mounted
→ unit refuses to start when storage is gone
With the inverted condition, postgresql started on empty bind-mount
after lazy-umount of /home/oqyude/External — the exact silent-data-loss
scenario R1.2 is supposed to prevent.
This is the canonical 'bug the test caught' case. Live test v6 on
sapphira (2026-10-10) demonstrated: with '!' the guard does nothing,
without '!' the guard fires correctly.
Lesson: always run a live test of the guard, don't trust nix eval alone
for systemd Condition* semantics — they're evaluated by systemd at
runtime, and '!' inverts the test.
Comprehensive batch addressing the 16-task backlog in
.agent/tasks/manifest.json. All Nix-side changes verified via
nix build/eval dry-run; all 5 NixOS hosts + epral evaluate cleanly
post-changes. No regressions.
Wave 1 (non-functional cleanup):
T1/A1 — configurations/mobile.nix:12: fix `import ../lib/xlib.nix`
(broken path) → `import ../lib/xlib`. Unblocks nixOnDroid
configurations.epral. R1.1 invariant.
T8/C3 — modules/containers/3x-ui.nix: remove `podman-update-3xui_app`
systemd service and commented timer. Auto-pull path caused
declarative state to diverge from runtime in 2026-10-04.
R1.5 invariant.
T13/D3 — modules/server/nginx.nix:368-371: remove dead
`networking.firewall.allowedTCPPorts = [80 443]`.
`firewall.enable = false` on sapphira (R1.3), so openFirewall
rules are no-op. Replace with R1.3 comment.
T6/C1 — .agent/decisions/notes/3x-ui-xray-26.9.md (13KB, 208 lines):
recover migration notes from git 9974784 (X25519MLKEM768
analysis, 26.7→26.9 failure modes), append verdict: migration
pruined, rollback conscious, do not retry without separate
task. R1.5 / C1.
T9/C4 — .agent/rules/project-rules.md: add R1.8 — Xray-core version is
state of 3x-ui panel, not Nix. Update trap entry for
3x-ui.nix:54 to reference R1.8.
T11/D1, T12/D2 — .agent/checkpoints.json + .agent/tasks/manifest.json:
verify R1.3 (router port-forwards 22/80/443/8443/22000) and
R1.4 (100.64.0.0 = Tailscale sapphira) wording already
satisfies acceptance criteria. Flip status pending → completed.
T4 (storage guard, FUNCTIONAL CHANGE):
New helper in lib/xlib/helpers.nix:
mkStorageGuard = xlib: {
RequiresMountsFor = [ xlib.dirs.server-home ];
ConditionPathIsMountPoint = [ "!${xlib.dirs.server-home}" ];
};
Applied to 13 systemd units via path-style override:
- modules/server/{postgresql,samba,homebox,gitea,navidrome,
syncthing,uptime-kuma,immich,nextcloud,calibre-web}.nix
- modules/containers/3x-ui.nix (podman-3xui_app)
- modules/containers/tape-rotation.nix (podman-taperotation-{backend,frontend})
Anchor: xlib.dirs.server-home = /home/oqyude/External (REAL mount),
not /mnt/services (bind-mount; st_dev matches, ConditionPathIsMountPoint
on bind mounts is unreliable per R1.2 note).
Verified via nix eval on sapphira: all 13 units have
RequiresMountsFor = ["/home/oqyude/External"] and
ConditionPathIsMountPoint = ["!/home/oqyude/External"].
Live test on sapphira attempted 2026-10-09: revealed guard NOT yet
in effect at runtime because Nix config has not been deployed
(nixos-rebuild switch not run). postgresql started despite External
being unmounted. Implementation correct, deployment pending user
action.
T7/C2 (read-only diag, no code change):
3x-ui version facts recorded in conversation (sapphira journal +
/var/lib/containers/storage/overlay/.../diff/app/bin/xray-linux-amd64):
- Active Xray: 26.7.28 (go1.26.5 linux/amd64) — R1.5 validated at runtime
- Stale binary: 26.9.30 (go1.27.1) — leftover from failed 26.9 migration
- Panel DB (x-ui.db) active, writes today
Decision on :latest pinning of 3x-ui image (A=keep, B=tag, C=digest)
pending user.
T3/A3 (nftables on otreca — config analysis + proposal):
Diagnostic attempted via ssh otreca-tailscale (100.64.1.0) and
otreca public (109.248.161.5:22): BOTH UNREACHABLE. Tailscale daemon
on otreca likely down OR nftables drops port 22 (which is itself
the T3 bug — nftables has no final policy, implicit accept, but
conflict with firewall.enable = true per R1.6).
Proposal written: .agent/decisions/proposals/vds-nftables-fix.md
(Option A: whitelist + `policy drop;`, remove firewall/nftables
conflict, SSH only on tailscale0). Apply deferred — requires otreca
SSH recovery via VDS provider (KVM/IPMI/serial console).
T5/B2 (backups documentation):
.agent/decisions/0002-backups-external.md (draft): catalog of what
is declared in Nix vs. what is external; awaiting answer to open
question 5.6 (where are backups, how are they verified).
T15/E2 (CI checks):
.ci/checks.sh (executable, ~140 lines) with 3 checks from
analysis-report.md §5:
- #1: no `:latest` in container images (with R1.5 whitelist
for 3x-ui). FAIL — 4 violations:
localhost/kokoro-tts:latest
ghcr.io/openhands/openhands:latest
docker.io/elizaroveugene/taperotation-backend:latest
docker.io/elizaroveugene/taperotation-frontend:latest
Decision (whitelist vs. pin) pending user.
- #2: nix flake check (skipped with --no-build).
- #7: secrets/ files match .sops.yaml path_regex. PASS.
T16/E3 (archive commented modules):
13 of 14 commented modules in modules/server/default.nix:37-50
existed as files. git mv them to archive/{server-modules,containers}/.
1 (stirling-pdf.nix) didn't exist; just removed the comment.
modules/server/default.nix:37-50 cleaned of 14 commented lines.
Added 3-line comment recording the archive date and reason.
Verified: nixosConfigurations.sapphira still evaluates.
Post-change state:
$ nix build .#nixosConfigurations.{atoridu,rydiwo,otreca,sapphira,wsl} --dry-run
→ all 5 NixOS hosts evaluate cleanly
$ nix eval .#nixOnDroidConfigurations.epral.config.system.stateVersion
→ "24.05"
Pending (user input required — not in this commit):
- T4 deploy: run `nixos-rebuild switch` on sapphira to activate guard
- T7: pick A/B/C for 3x-ui :latest pinning
- T3: recover otreca SSH via VDS provider, then apply Option A
- T10/C5: decide fate of reality443Forwarding
- T5: answer 5.6 about backup location/verification
- T15: whitelist or pin 4 :latest images
Untracked files NOT committed (in .gitignore):
.temp/t4-live-test*.sh, .temp/cleanup-*.sh — throwaway test scripts
from T4 live test attempts. Preserved locally for reference; see
AGENTS.md convention ("Создавать `.temp/` в корне проекта — Для
временных файлов агента. Всегда в `.gitignore`").
Also untracked, committed:
.agent/reviews/2026-10-10-review-dev-diff-vs-16644fc.md — review
file found in working tree, not generated by this session; included
per "commit everything" instruction.
The path '/home/<user>/.config/opencode/server.env' was duplicated
between users.nix (sops materialization) and home/modules/opencode.nix
(programs.opencode.web.environmentFile). Drift between the two was a
silent auth-bypass vector: if one moved, the systemd unit would either
fail to find OPENCODE_SERVER_PASSWORD or skip EnvironmentFile entirely.
Single source in lib/xlib/dirs.nix; both call sites now read from it.
oqyude