release: v1.12.0 (promote develop to main via merge)

Author: VijitSingh97

CHANGELOG.md

@@ -7,6 +7,50 @@ All notable changes to RigForge are documented here. The format is based on
 
 ## [Unreleased]
 
+## [1.12.0] - 2026-07-19
+
+The pithead#597 producer release: the control-upgrade `/status` contract a one-click worker
+upgrade can poll without string-matching, a fail-closed anti-beacon throttle, corrected
+security docs, and the remote-upgrade chain codified as a repeatable real-hardware e2e phase.
+
+### Added
+
+- **`control-upgrade` status contract: `started` / `noop` / `throttled` + version echo (#320).**
+  Consumer findings from pithead's one-click worker upgrade (pithead#597): the oneshot now writes a
+  non-terminal `started` record the moment it claims a staged intent (so a poller can tell mid-build
+  from queued, and a `started` that outlives the oneshot means the run was lost); the already-on-target
+  refusal reports `noop` and the throttle window reports `throttled` instead of collapsing into
+  `failed`; a successful upgrade's `reason` names the landed version. Additive to the ADR 0002 D3/D6
+  model — D6 amended in place, contract documented in Operations › Writable control path.
+- **`tests/e2e-real.sh` grows an `upgrade` phase (#322).** Codifies the real-hardware
+  control-upgrade run that caught both real bugs in this chain (#308's dubious-ownership silent git
+  death, #318's origin/HEAD refusal) — invisible to the unit suite by design, since it stubs git. A
+  noop leg proves the wire → path unit → root oneshot → `/status` round trip; a rollback leg forges
+  a local tag unreachable from `origin/main` and asserts the D10 refusal rolls back with the tree,
+  `VERSION`, and miner intact and the D6 throttle stamped; an opt-in `E2E_UPGRADE_TARGET` leg drives
+  a real forward upgrade. Runs inside `all` between `control` and `perf`, with the same
+  snapshot-and-revert guarantees. Also the producer half of pithead#597's cross-repo gate.
+
+### Fixed
+
+- **The anti-beacon throttle fails CLOSED on an unusable state dir (#321).** It used to proceed
+  unthrottled (with a warn) when it couldn't take its lock — inverting the guard's whole purpose on
+  exactly the kind of degraded box an attacker might arrange. Now the upgrade is refused with a
+  `failed` status whose reason names the throttle state, distinct from `throttled` so a consumer
+  doesn't retry-later forever against a broken rig.
+- **Disabling the control path re-stops the receiver explicitly.** On the v1.12.0 pre-tag gate,
+  a loaded systemctl transiently dropped the stop half of `disable --now`: the unit files were
+  removed but the running receiver was left orphaned — a live write surface on a rig whose config
+  said control was off. The disable path now issues one explicit re-stop (no-op when the stop
+  landed). The `control` e2e phase also gained the polled receiver-up gate the `upgrade` phase
+  ships with — its single-shot check hit the same pinned-CPU cold-start race on the gate run.
+- **Security-doc drift on the remote-upgrade trust model (#321).** SECURITY.md claimed the remote
+  path is "verified by SHA256SUMS hash only" — it never was; the real anchor is the ADR 0002 D5
+  design (git checkout of the release tag over TLS, `origin/main` ancestry, immutable releases), and
+  the doc now says so. ADR 0002 D8's "a flock serializes concurrent triggers" corrected in place:
+  the only flock guards the throttle stamp; serialization is `Type=oneshot` + newest-wins spool
+  draining.
+
 ## [1.11.2] - 2026-07-18
 
 Consistency-audit patch: CLI-surface and docs cleanups, plus a real fix to the remote-upgrade

RELEASING.md

@@ -29,6 +29,7 @@ promoted to `main` and tagged. The steps below build the release commit on `deve
    sudo reboot                             # HugePages (1G + GRUB cmdline) take effect on boot; reconnect
    sudo bash tests/e2e-real.sh verify      # doctor (HugePages/MSR/governor/service) + bench (real H/s) + a short tune + a live auto-tune pass
    sudo bash tests/e2e-real.sh control     # the writable control path (#236) against real systemd: enable, POST a change, poll to applied, revert
+   sudo bash tests/e2e-real.sh upgrade     # the remote-upgrade chain (#308/#322) with REAL git: noop + refused-tag rollback legs, revert (opt-in forward leg: E2E_UPGRADE_TARGET=vX.Y.Z)
    sudo bash tests/e2e-real.sh perf        # offline bench vs the committed per-host baseline + best-ever history (the release perf gate)
    sudo bash tests/e2e-real.sh teardown    # uninstall + assert a clean revert
    ```

SECURITY.md

@@ -121,13 +121,16 @@ the key-custody burden wasn't worth it for this project's users, who deploy from
 that trade ever changes (a real download-based user base), the signing machinery lives in the
 git history of `.github/workflows/release.yml` (#137, #205).
 
-The same trust boundary governs the planned remote-upgrade path (`control-upgrade`, #308 — opt-in,
-off by default). There a rig would fetch a release and run it as root unattended, verified by
-`SHA256SUMS` hash only. That path removes the human who would otherwise eyeball a bad release, so it
-inherits the origin gap above as a deliberately accepted risk: a full GitHub-account compromise could
-push malicious root code across a fleet. We accept that rather than plan around a compromise of
-GitHub itself — the proportionate mitigation is hardening the account (hardware-key 2FA, tight
-release and branch-protection permissions), not re-adding release signing.
+The same trust boundary governs the remote-upgrade path (`control-upgrade`, #308 — opt-in, off by
+default, [ADR 0002](docs/adr/0002-remote-worker-upgrade.md)). There a rig fetches a release and runs
+it as root unattended — but not via the release bundles or `SHA256SUMS` at all: the fetch is a
+**git checkout of the release tag over TLS to GitHub**, gated on the tag's commit being reachable
+from `origin/main` and on GitHub's immutable releases locking the tag→commit binding. Checksums play
+no role on that path (ADR 0002 D5). It removes the human who would otherwise eyeball a bad release,
+so it inherits the origin gap above as a deliberately accepted risk: a full GitHub-account
+compromise could push malicious root code across a fleet. We accept that rather than plan around a
+compromise of GitHub itself — the proportionate mitigation is hardening the account (hardware-key
+2FA, tight release and branch-protection permissions), not re-adding release signing.
 
 For the strongest guarantee available today, deploy from a git checkout at the release tag —
 the commit hash is the integrity anchor, and the XMRig source it builds is itself pinned and

VERSION

@@ -1 +1 @@
-1.11.2
+1.12.0

docs/adr/0002-remote-worker-upgrade.md

@@ -33,7 +33,7 @@ The receiver (`util/control-server.py`) gains a `POST /upgrade` distinct from `/
 
 ### D3. Unprivileged receiver stages only; the privileged oneshot fetches and runs
 
-As in ADR 0001 D2, the network-facing receiver stays unprivileged (`DynamicUser`): it authenticates, checks the flag, and **stages** the request; it never fetches or runs code itself. The privileged path-triggered oneshot does the fetch-and-upgrade through the existing gated flow. The response is `202 Accepted` with a change id; the consumer polls `/status` for the terminal outcome.
+As in ADR 0001 D2, the network-facing receiver stays unprivileged (`DynamicUser`): it authenticates, checks the flag, and **stages** the request; it never fetches or runs code itself. The privileged path-triggered oneshot does the fetch-and-upgrade through the existing gated flow. The response is `202 Accepted` with a change id; the consumer polls `/status` to a terminal outcome (a non-terminal `started` appears first while the oneshot runs — #320, see D6).
 
 ### D4. Dashboard-supplied target, bounded by monotonic + reachability guards
 
@@ -43,7 +43,7 @@ The staged body is tiny: `{"version": "vX.Y.Z"}`, and it **is** the target. The
 - **Reachable-from-main** — refuse a commit that is not an ancestor of `origin/main` (D10; amended by #318).
 - **Immutable releases + tag protection** (D10) make the tag→commit binding a platform guarantee.
 
-The rig still `git`-fetches the tag directly from GitHub — the code has to come from somewhere, and that fetch happens only on an actual upgrade (rare, throttled), not on every trigger — but it makes no separate version-check dial. Anything that fails a guard is terminal `failed`. This trades away rig-*independent* confirmation of "is this THE latest" (the dashboard, over Tor, is the deriver) for privacy, fewer moving parts, and no fleet rate-limit coupling; the anti-rollback + reachability guards are what make trusting a supplied target safe.
+The rig still `git`-fetches the tag directly from GitHub — the code has to come from somewhere, and that fetch happens only on an actual upgrade (rare, throttled), not on every trigger — but it makes no separate version-check dial. Anything that fails a guard ends terminal: `failed`, except the two benign refusals distinguished since #320 — `noop` (already on the target) and `throttled` (inside the D6 window). This trades away rig-*independent* confirmation of "is this THE latest" (the dashboard, over Tor, is the deriver) for privacy, fewer moving parts, and no fleet rate-limit coupling; the anti-rollback + reachability guards are what make trusting a supplied target safe.
 
 ### D5. Trust model: hash-only, git-tag checkout, no signing (settled)
 
@@ -55,6 +55,8 @@ GitHub is therefore accepted as the trust root. The residual risk is explicit: a
 
 `control-upgrade` is fail-closed and every step is rollback-guarded, reusing the upgrade flow's existing rebuild-only-if-the-pin-changed logic. A **pre-dial throttle stamp** on the rig bounds how often it will reach out to GitHub, so a looping or hostile consumer cannot turn the fleet into a beacon or a request amplifier. Outcomes use the same `/status` surface as control-apply, extended with a **`failed`** terminal (with reason) alongside `applied` / `rolled_back`, so a pre-apply refusal (non-latest, throttled, fetch/build error) surfaces a clear terminal result to the consumer's poll rather than hanging in a non-terminal state.
 
+*Amended (#320, consumer feedback from pithead #597):* the status vocabulary grew three additive members so a poller never has to string-match `reason`. A non-terminal **`started`** is written the moment the oneshot claims the intent (D8 move done), so "mid-run" and "oneshot died mid-run" are distinguishable from "queued"; **`noop`** replaces `failed` for the already-on-target refusal (idempotent, not an error); **`throttled`** replaces `failed` for the D6 throttle refusal (retry-later, not an error). `applied`'s `reason` echoes the landed version. All other refusals stay `failed`.
+
 ### D7. Auth and source-pinning inherited from ADR 0001 D4/D9
 
 `/upgrade` reuses the control path's Bearer `ACCESS_TOKEN` and the `api_allow_from` source pin (which D1 already requires for `control: enabled`), and the same nftables scoping (#142). The stack host is the only trusted writer; the miner never accepts an upgrade trigger from a miner-advertised or arbitrary host. The cleartext-token-on-LAN posture (ADR 0001 D9) applies unchanged — the mining LAN is the trust boundary.
@@ -71,7 +73,7 @@ The **root builder oneshot cannot be meaningfully sandboxed**, and the earlier D
 
 It comes from the **verb**, which treats the staged file as untrusted input: `systemd.path` validates nothing, so the oneshot's first act is to `rename()` the intent into a root-owned `processing/` dir the `DynamicUser` cannot write — freezing it against any swap — and *then* refuse a symlink and parse it as a claim (a single whitelisted field, `version` matching `^v[0-9]+\.[0-9]+\.[0-9]+$`, never sourced/evaled). Doing the move first is deliberate: checking for a symlink *before* the move would be a TOCTOU (the `DynamicUser` owns the spool and could swap the file in the window); after the move the file lives in a root-only dir and can't be swapped. That plus the D4/D6/D10 validation (target-newer-and-reachable bounds, throttle, anti-rollback) and a **rollback that reverts a failed forward build** — not just a failed liveness check, so a build error never leaves the tree pinned to an unbuilt release — is where this path is actually defended.
 
-`systemd.path` validates nothing about the staged file — it only watches for its existence — so the spool file is fully attacker-controlled input from the root unit's perspective (the same "observed content is data, not a command" boundary, in code). Defenses: the receiver writes atomically (`rename()` into place, `StateDirectoryMode=0700`); the root oneshot's **first action is `rename()` the trigger into a root-owned dir the `DynamicUser` cannot write** (atomically removing any swap/symlink/TOCTOU window), reads it once with `O_NOFOLLOW`, and parses it as a claim — a single whitelisted field (tag matching `^v[0-9]+\.[0-9]+\.[0-9]+$`), never sourced/evaled, never passed unquoted to a shell, the repo URL hardcoded in the unit. A `flock` serializes concurrent triggers.
+`systemd.path` validates nothing about the staged file — it only watches for its existence — so the spool file is fully attacker-controlled input from the root unit's perspective (the same "observed content is data, not a command" boundary, in code). Defenses: the receiver writes atomically (`rename()` into place, `StateDirectoryMode=0700`); the root oneshot's **first action is `rename()` the trigger into a root-owned dir the `DynamicUser` cannot write** (atomically removing any swap/symlink/TOCTOU window), reads it once with `O_NOFOLLOW`, and parses it as a claim — a single whitelisted field (tag matching `^v[0-9]+\.[0-9]+\.[0-9]+$`), never sourced/evaled, never passed unquoted to a shell, the repo URL hardcoded in the unit. Concurrent triggers are serialized by systemd itself: the `Type=oneshot` unit runs one instance at a time, and each run drains the spool newest-wins, so superseded intents are dropped rather than replayed. *(Amended #321: an earlier version of this decision claimed "a `flock` serializes concurrent triggers" — the only `flock` on this path guards the D6 throttle stamp, released before the build starts; the verb itself is serialized by the oneshot semantics above. If oneshot semantics are ever not enough, take a verb-scoped lock spanning fetch/build.)*
 
 ### D9. Protect hashrate by spatial build isolation, not just priority (the no-perf-impact requirement)
 
@@ -100,7 +102,7 @@ Low-cost integrity controls that need no standing signing key, shoring up the Gi
 - New opt-in surface, off by default, fail-closed. A rig that does not enable `control_upgrade` carries zero new capability; a rig on `control` alone is unchanged.
 - Residual risk accepted: GitHub-account/release compromise → fleet-wide root RCE (D5). Mitigation is account hardening, not signing.
 - Producer for pithead #597. Backwards compatible (a new MINOR capability). It must ship in a RigForge release before pithead #597 can run end-to-end; pithead #596 (version badge) has no RigForge dependency and ships first.
-- Consumer contract: `POST :8082/upgrade {"version":"vX.Y.Z"}` → `202 Accepted` + change id → poll `:8082/status` for `applied` / `rolled_back` / `failed`. The host never trusts a host/port/token from the intent; the rig bounds the dashboard-supplied target with monotonic anti-rollback + reachable-from-main guards (D4/D10) rather than making its own version-check dial.
+- Consumer contract: `POST :8082/upgrade {"version":"vX.Y.Z"}` → `202 Accepted` + change id → poll `:8082/status` for a non-terminal `started`, then terminal `applied` (reason names the landed version) / `rolled_back` / `noop` (already on target) / `throttled` (retry after the window) / `failed` (#320 amendment, D6). The host never trusts a host/port/token from the intent; the rig bounds the dashboard-supplied target with monotonic anti-rollback + reachable-from-main guards (D4/D10) rather than making its own version-check dial.
 
 ## Open questions (resolved 2026-07-17 — see Resolution above)
 

docs/operations.md

@@ -601,6 +601,13 @@ and health-gates the swap with rollback. It refuses anything that isn't a real r
 the running one. Default off, and never on just because `control` is. The threat model and
 decisions are in [ADR 0002](adr/0002-remote-worker-upgrade.md).
 
+Polling an upgrade works like polling a config change, with a slightly richer vocabulary (#320):
+`GET :8082/status?change_id=<id>` serves a non-terminal `started` once the root oneshot has claimed
+the intent (so a poller can tell "mid-build" from "queued" — and a `started` that outlives the
+oneshot means the run was lost), then one terminal outcome: `applied` (the `reason` names the
+landed version), `rolled_back`, `noop` (already on the requested version — idempotent, not an
+error), `throttled` (inside the anti-beacon window — retry later), or `failed` (with `reason`).
+
 `"control": "disabled"` + `apply` removes the units cleanly; `uninstall` removes them too. Linux-only.
 The design and the decisions behind it are recorded in
 [ADR 0001](../docs/adr/0001-writable-worker-config-control-path.md).

docs/pithead-integration.md

@@ -188,8 +188,11 @@ and [ADR 0001](adr/0001-writable-worker-config-control-path.md).
 recent* change's outcome, which a concurrent change (another dashboard edit, a local `apply`, an
 autotune restart) can step on between your `POST` and your poll. To avoid that race, poll
 `GET :8082/status?change_id=<the 16-hex id from the 202>` — it returns *that* change's recorded
-outcome (`applied`/`rejected`/`rolled_back`/`failed` + `reason`/`backup`/`changed_keys`/`warnings`), or `404`
-if it isn't among the last ~20 recorded. Same bearer auth as `/status`. Pair it with
+outcome (for a config change: `applied`/`rejected`/`rolled_back`/`failed` +
+`reason`/`backup`/`changed_keys`/`warnings`), or `404` if it isn't among the last ~20 recorded. Same
+bearer auth as `/status`. The same endpoint serves remote-*upgrade* change ids with a richer
+vocabulary (non-terminal `started`, terminal `noop`/`throttled` alongside the above — #320); see
+[Operations › Remote upgrade](operations.md#writable-control-path-opt-in). Pair it with
 `config_meta.revision` on the read feed to confirm the effective config actually moved.
 
 **Prefill from a live read (`rigforge.config`).** The enriched feed exposes the rig's current