Skip to content

rig-lock: pre-create /var/lock/rig-e2e.lock world-writable so a non-root reserve can't lock out the root e2e rig_lock #242

Description

@VijitSingh97

Why

The shared rig-lock (#183/#184, also honored by pithead #431) has a boot-order footgun that bit a real rollout on 2026-07-11.

/var/lock/run/lock, a sticky, world-writable tmpfs that is cleared on every reboot. The kernel has fs.protected_regular=2 (Ubuntu 24.04 default). Consequence: whoever creates /var/lock/rig-e2e.lock first after a reboot owns it — and if that's a non-root flock (a manual reserve, exactly what ~/README.md invites), the file lands -rw-rw-r-- <user>:<user>, and then the root e2e-real.sh rig_lock's exec 9>"$LOCK" fails with EACCES (protected_regular refuses a root O_CREAT-write of a file it doesn't own in a sticky world-writable dir). The perf gate then aborts (rc=1, "Permission denied") until someone rms the file.

Reproduced on miner-2:

$ flock -n /run/lock/rl-test true       # as vijit -> -rw-rw-r-- vijit vijit
$ sudo bash -c 'exec 9>/run/lock/rl-test'   # root -> "Permission denied"
$ sudo install -m 666 /dev/null /run/lock/rl-test2
$ sudo bash -c 'exec 9>/run/lock/rl-test2'  # root -> OK
$ sysctl fs.protected_regular              # = 2

rig_lock already chmod 666s the file after it opens it — so once the root e2e suite has run, it's shared-safe. The trap is purely "non-root got there first after a reboot."

Fix

Ensure the lock file always exists as root:root 0666 before anyone flocks it, so a non-root reserve joins an existing good file instead of creating a bad one:

  • Ship a tmpfiles.d snippet (installed by setup on Linux, next to the other units): f /var/lock/rig-e2e.lock 0666 root root -. systemd-tmpfiles recreates it at every boot, ahead of any manual reserve.
  • Cross-repo: pithead #431 uses the same path; whichever stack provisions the box installs it (or both ship the identical snippet — idempotent).

Interim mitigation already applied: ~/README.md on all rigs now says reserve as root (sudo flock ...) and how to recover (sudo rm the stale file when unheld).

Acceptance

  • After a reboot, /var/lock/rig-e2e.lock exists root:root 0666 before any run.
  • A non-root flock reserve followed by a root e2e-real.sh perf both succeed (no EACCES).
  • make lint + make test pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    infraDeployment, packaging, releasestestingTests, CI, and test infrastructure

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions