feat(control): richer /status contract for control-upgrade — started, noop, throttled, version echo (#320) (#323)

* feat(control): richer /status contract for control-upgrade — started, noop, throttled, version echo (#320)

Consumer findings from pithead#597 against the v1.11.2 contract:

- Write one non-terminal {status:"started"} record right after the D8
  claim, so a poller can tell mid-build from queued — and a started
  record that outlives the oneshot means the run was lost (poll-forever
  otherwise).
- already-on-target reports noop (idempotent, not red on a dashboard);
  the throttle window reports throttled (retry-later) — both were an
  undifferentiated failed distinguishable only by prose reason.
- applied's reason echoes the landed version, so the consumer no longer
  cross-reads :8081/2/summary and hopes no second change raced it.

Additive to the ADR 0002 D3/D6 model (D6 amended in place); documented
in Operations › Writable control path. Unit suite covers the started
mid-run record (incl. the changes/<cid> index), the noop and throttled
statuses, and the version echo.

Closes #320

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(control): sweep the remaining status-vocabulary drift after #320

Docs-consistency review before merge: ADR 0002 D4 still claimed every
guard failure is terminal failed (noop/throttled exist now), D3 predated
the started record, _control_status's comment said 'outcome' only,
control-server.py's docstring omitted /upgrade and the non-terminal
record (and its 202 note read as if already-on-target were a refusal),
and pithead-integration.md's ?change_id status list could be misread as
the endpoint's full vocabulary for upgrade change ids.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: VijitSingh97

CHANGELOG.md

@@ -7,6 +7,16 @@ All notable changes to RigForge are documented here. The format is based on
 
 ## [Unreleased]
 
+### Added
+
+- **`control-upgrade` status contract: `started` / `noop` / `throttled` + version echo (#320).**
+  Consumer findings from pithead's one-click worker upgrade (pithead#597): the oneshot now writes a
+  non-terminal `started` record the moment it claims a staged intent (so a poller can tell mid-build
+  from queued, and a `started` that outlives the oneshot means the run was lost); the already-on-target
+  refusal reports `noop` and the throttle window reports `throttled` instead of collapsing into
+  `failed`; a successful upgrade's `reason` names the landed version. Additive to the ADR 0002 D3/D6
+  model — D6 amended in place, contract documented in Operations › Writable control path.
+
 ## [1.11.2] - 2026-07-18
 
 Consistency-audit patch: CLI-surface and docs cleanups, plus a real fix to the remote-upgrade

docs/adr/0002-remote-worker-upgrade.md

@@ -33,7 +33,7 @@ The receiver (`util/control-server.py`) gains a `POST /upgrade` distinct from `/
 
 ### D3. Unprivileged receiver stages only; the privileged oneshot fetches and runs
 
-As in ADR 0001 D2, the network-facing receiver stays unprivileged (`DynamicUser`): it authenticates, checks the flag, and **stages** the request; it never fetches or runs code itself. The privileged path-triggered oneshot does the fetch-and-upgrade through the existing gated flow. The response is `202 Accepted` with a change id; the consumer polls `/status` for the terminal outcome.
+As in ADR 0001 D2, the network-facing receiver stays unprivileged (`DynamicUser`): it authenticates, checks the flag, and **stages** the request; it never fetches or runs code itself. The privileged path-triggered oneshot does the fetch-and-upgrade through the existing gated flow. The response is `202 Accepted` with a change id; the consumer polls `/status` to a terminal outcome (a non-terminal `started` appears first while the oneshot runs — #320, see D6).
 
 ### D4. Dashboard-supplied target, bounded by monotonic + reachability guards
 
@@ -43,7 +43,7 @@ The staged body is tiny: `{"version": "vX.Y.Z"}`, and it **is** the target. The
 - **Reachable-from-main** — refuse a commit that is not an ancestor of `origin/main` (D10; amended by #318).
 - **Immutable releases + tag protection** (D10) make the tag→commit binding a platform guarantee.
 
-The rig still `git`-fetches the tag directly from GitHub — the code has to come from somewhere, and that fetch happens only on an actual upgrade (rare, throttled), not on every trigger — but it makes no separate version-check dial. Anything that fails a guard is terminal `failed`. This trades away rig-*independent* confirmation of "is this THE latest" (the dashboard, over Tor, is the deriver) for privacy, fewer moving parts, and no fleet rate-limit coupling; the anti-rollback + reachability guards are what make trusting a supplied target safe.
+The rig still `git`-fetches the tag directly from GitHub — the code has to come from somewhere, and that fetch happens only on an actual upgrade (rare, throttled), not on every trigger — but it makes no separate version-check dial. Anything that fails a guard ends terminal: `failed`, except the two benign refusals distinguished since #320 — `noop` (already on the target) and `throttled` (inside the D6 window). This trades away rig-*independent* confirmation of "is this THE latest" (the dashboard, over Tor, is the deriver) for privacy, fewer moving parts, and no fleet rate-limit coupling; the anti-rollback + reachability guards are what make trusting a supplied target safe.
 
 ### D5. Trust model: hash-only, git-tag checkout, no signing (settled)
 
@@ -55,6 +55,8 @@ GitHub is therefore accepted as the trust root. The residual risk is explicit: a
 
 `control-upgrade` is fail-closed and every step is rollback-guarded, reusing the upgrade flow's existing rebuild-only-if-the-pin-changed logic. A **pre-dial throttle stamp** on the rig bounds how often it will reach out to GitHub, so a looping or hostile consumer cannot turn the fleet into a beacon or a request amplifier. Outcomes use the same `/status` surface as control-apply, extended with a **`failed`** terminal (with reason) alongside `applied` / `rolled_back`, so a pre-apply refusal (non-latest, throttled, fetch/build error) surfaces a clear terminal result to the consumer's poll rather than hanging in a non-terminal state.
 
+*Amended (#320, consumer feedback from pithead #597):* the status vocabulary grew three additive members so a poller never has to string-match `reason`. A non-terminal **`started`** is written the moment the oneshot claims the intent (D8 move done), so "mid-run" and "oneshot died mid-run" are distinguishable from "queued"; **`noop`** replaces `failed` for the already-on-target refusal (idempotent, not an error); **`throttled`** replaces `failed` for the D6 throttle refusal (retry-later, not an error). `applied`'s `reason` echoes the landed version. All other refusals stay `failed`.
+
 ### D7. Auth and source-pinning inherited from ADR 0001 D4/D9
 
 `/upgrade` reuses the control path's Bearer `ACCESS_TOKEN` and the `api_allow_from` source pin (which D1 already requires for `control: enabled`), and the same nftables scoping (#142). The stack host is the only trusted writer; the miner never accepts an upgrade trigger from a miner-advertised or arbitrary host. The cleartext-token-on-LAN posture (ADR 0001 D9) applies unchanged — the mining LAN is the trust boundary.
@@ -100,7 +102,7 @@ Low-cost integrity controls that need no standing signing key, shoring up the Gi
 - New opt-in surface, off by default, fail-closed. A rig that does not enable `control_upgrade` carries zero new capability; a rig on `control` alone is unchanged.
 - Residual risk accepted: GitHub-account/release compromise → fleet-wide root RCE (D5). Mitigation is account hardening, not signing.
 - Producer for pithead #597. Backwards compatible (a new MINOR capability). It must ship in a RigForge release before pithead #597 can run end-to-end; pithead #596 (version badge) has no RigForge dependency and ships first.
-- Consumer contract: `POST :8082/upgrade {"version":"vX.Y.Z"}` → `202 Accepted` + change id → poll `:8082/status` for `applied` / `rolled_back` / `failed`. The host never trusts a host/port/token from the intent; the rig bounds the dashboard-supplied target with monotonic anti-rollback + reachable-from-main guards (D4/D10) rather than making its own version-check dial.
+- Consumer contract: `POST :8082/upgrade {"version":"vX.Y.Z"}` → `202 Accepted` + change id → poll `:8082/status` for a non-terminal `started`, then terminal `applied` (reason names the landed version) / `rolled_back` / `noop` (already on target) / `throttled` (retry after the window) / `failed` (#320 amendment, D6). The host never trusts a host/port/token from the intent; the rig bounds the dashboard-supplied target with monotonic anti-rollback + reachable-from-main guards (D4/D10) rather than making its own version-check dial.
 
 ## Open questions (resolved 2026-07-17 — see Resolution above)
 

docs/operations.md

@@ -601,6 +601,13 @@ and health-gates the swap with rollback. It refuses anything that isn't a real r
 the running one. Default off, and never on just because `control` is. The threat model and
 decisions are in [ADR 0002](adr/0002-remote-worker-upgrade.md).
 
+Polling an upgrade works like polling a config change, with a slightly richer vocabulary (#320):
+`GET :8082/status?change_id=<id>` serves a non-terminal `started` once the root oneshot has claimed
+the intent (so a poller can tell "mid-build" from "queued" — and a `started` that outlives the
+oneshot means the run was lost), then one terminal outcome: `applied` (the `reason` names the
+landed version), `rolled_back`, `noop` (already on the requested version — idempotent, not an
+error), `throttled` (inside the anti-beacon window — retry later), or `failed` (with `reason`).
+
 `"control": "disabled"` + `apply` removes the units cleanly; `uninstall` removes them too. Linux-only.
 The design and the decisions behind it are recorded in
 [ADR 0001](../docs/adr/0001-writable-worker-config-control-path.md).

docs/pithead-integration.md

@@ -188,8 +188,11 @@ and [ADR 0001](adr/0001-writable-worker-config-control-path.md).
 recent* change's outcome, which a concurrent change (another dashboard edit, a local `apply`, an
 autotune restart) can step on between your `POST` and your poll. To avoid that race, poll
 `GET :8082/status?change_id=<the 16-hex id from the 202>` — it returns *that* change's recorded
-outcome (`applied`/`rejected`/`rolled_back`/`failed` + `reason`/`backup`/`changed_keys`/`warnings`), or `404`
-if it isn't among the last ~20 recorded. Same bearer auth as `/status`. Pair it with
+outcome (for a config change: `applied`/`rejected`/`rolled_back`/`failed` +
+`reason`/`backup`/`changed_keys`/`warnings`), or `404` if it isn't among the last ~20 recorded. Same
+bearer auth as `/status`. The same endpoint serves remote-*upgrade* change ids with a richer
+vocabulary (non-terminal `started`, terminal `noop`/`throttled` alongside the above — #320); see
+[Operations › Remote upgrade](operations.md#writable-control-path-opt-in). Pair it with
 `config_meta.revision` on the read feed to confirm the effective config actually moved.
 
 **Prefill from a live read (`rigforge.config`).** The enriched feed exposes the rig's current

rigforge.sh

@@ -3812,7 +3812,8 @@ _control_do_apply() {
     _wait_miner_live "${CONTROL_LIVE_TRIES:-20}"
 }
 
-# Record the outcome for the receiver's GET /status (mode 644 so the DynamicUser server reads it back).
+# Record a status record for the receiver's GET /status (mode 644 so the DynamicUser server reads it
+# back) — a terminal outcome, or control_upgrade's non-terminal `started` marker (#320).
 _control_status() { # <status-file> <status> <cid> <keys-csv> <reason> <backup>
     local f="$1" cid="$3" body cdir
     mkdir -p "$(dirname "$f")"
@@ -3988,6 +3989,12 @@ control_upgrade() {
         warn "control-upgrade: $cid was a symlink — refused."
         return 0
     fi
+    # #320: one NON-terminal record now that the intent is claimed (D8 move done, nothing can swap it).
+    # Between 202 Accepted and the terminal write the poller previously saw only the PREVIOUS change's
+    # outcome — indistinguishable from "oneshot died before writing status". `started` is overwritten
+    # by this run's terminal record below; a poller that still reads it after the oneshot exited knows
+    # the run was lost.
+    _control_status "$status" started "$cid" version "" ""
     # Strict field whitelist (D4): exactly {"version":"vX.Y.Z"}. Never sourced/evaled; anything else refused.
     target=$(jq -r 'if (type == "object" and (keys | sort) == ["version"] and (.version | type == "string")) then .version else empty end' "$staged" 2>/dev/null)
     rm -f "$staged"
@@ -4003,7 +4010,8 @@ control_upgrade() {
     if [ -n "$installed" ]; then
         local want="${target#v}" highest
         if [ "$want" = "$installed" ]; then
-            _control_status "$status" failed "$cid" version "already on $target — nothing to upgrade" ""
+            # #320: an idempotent no-op, not a failure — the rig is exactly where the operator wants it.
+            _control_status "$status" noop "$cid" version "already on $target — nothing to upgrade" ""
             log "control-upgrade: already on $target."
             return 0
         fi
@@ -4016,15 +4024,17 @@ control_upgrade() {
     fi
     # D6 throttle: bound how often a rig fetches/rebuilds.
     if ! _control_upgrade_throttle_ok "$state"; then
-        _control_status "$status" failed "$cid" version "throttled — too soon since the last upgrade attempt" ""
+        # #320: distinct from failed — the consumer's right move is retry-later, not alarm.
+        _control_status "$status" throttled "$cid" version "throttled — too soon since the last upgrade attempt" ""
         warn "control-upgrade: throttled."
         return 0
     fi
     old_tag=$(git -C "$SCRIPT_DIR" -c safe.directory="$SCRIPT_DIR" describe --tags --exact-match 2>/dev/null || git -C "$SCRIPT_DIR" -c safe.directory="$SCRIPT_DIR" rev-parse HEAD 2>/dev/null || true)
     log "control-upgrade: $cid -> $target (from ${old_tag:-unknown}); fetching + building..."
     local RIGFORGE_CONFIG_SOURCE=control RIGFORGE_CONFIG_CHANGE_ID="$cid"
     if _control_upgrade_do "$target" && _wait_miner_live "${CONTROL_LIVE_TRIES:-20}"; then
-        _control_status "$status" applied "$cid" version "" ""
+        # #320: echo the landed version — the poller shouldn't have to cross-read the miner API for it.
+        _control_status "$status" applied "$cid" version "upgraded to $target" ""
         log "control-upgrade: $cid applied — now on $target."
     else
         # ANY forward failure rolls back — a reachability/build error OR the miner not returning to a live

tests/run.sh

@@ -6166,6 +6166,8 @@ cu_run() { # <staged-json|""> <installed-version> <do:ok|fail|down> -> status.js
 st() { printf '%s' "$1" | jq -r .status 2>/dev/null; }
 s="$(cu_run '{"version":"v9.9.9"}' "1.0.0" ok)"
 assert_eq "upgrade applied on a newer, buildable release" "$(st "$s")" "applied"
+# #320: the applied record must say which version landed — no cross-reading the miner API for it.
+assert_contains "applied reason echoes the landed version (#320)" "$s" "upgraded to v9.9.9"
 s="$(cu_run '{"version":"v9.9.9"}' "1.0.0" down)"
 assert_eq "built but miner stays down -> rolled_back" "$(st "$s")" "rolled_back"
 # #308 security review (HIGH): a build failure AFTER checkout must roll the tree back to the prior
@@ -6180,7 +6182,10 @@ s="$(cu_run '{"version":"v1.0.0"}' "2.0.0" ok)"
 assert_eq "downgrade refused -> failed (never built)" "$(st "$s")" "failed"
 assert_contains "downgrade reason names anti-rollback" "$s" "not newer"
 s="$(cu_run '{"version":"v1.0.0"}' "1.0.0" ok)"
-assert_contains "same version refused (already on)" "$s" "already on v1.0.0"
+# #320: already-on-target is an idempotent no-op, not a failure — a dashboard must not show red
+# for a rig sitting exactly where the operator wants it.
+assert_eq "same version -> noop, not failed (#320)" "$(st "$s")" "noop"
+assert_contains "noop reason still says already on" "$s" "already on v1.0.0"
 s="$(cu_run '{"version":"garbage"}' "1.0.0" ok)"
 assert_contains "malformed target refused, never run" "$s" "malformed"
 s="$(cu_run '{"version":"v9.9.9","evil":"x"}' "1.0.0" ok)"
@@ -6302,7 +6307,41 @@ sThr="$(
     PATH="$STUBS:$PATH" control_upgrade >/dev/null 2>&1
     cat "$cuThr/state/status.json" 2>/dev/null
 )"
-assert_contains "control_upgrade within the throttle window -> failed(throttled) (#308)" "$sThr" "throttled"
+assert_eq "control_upgrade within the throttle window -> status throttled (#308/#320)" "$(st "$sThr")" "throttled"
+assert_contains "throttled reason says why" "$sThr" "too soon"
+
+# #320: between the D8 claim and the terminal outcome the verb writes ONE non-terminal `started`
+# record, so a poller can tell "mid-run" (and "oneshot died mid-run": started never superseded)
+# from "queued, path unit hasn't fired" (previous change's terminal record still served).
+cuSt=$(mktemp -d "$SANDBOX/cust.XXXXXX")
+mkdir -p "$cuSt/state/spool"
+printf '1.0.0' >"$cuSt/VERSION"
+printf '{"pools":[{"url":"h:3333"}]}\n' >"$cuSt/config.json"
+printf '{"version":"v9.9.9"}\n' >"$cuSt/state/spool/upgrade-abc123def4567890.json"
+(
+    source "$SCRIPT"
+    OS_TYPE=Linux
+    SCRIPT_DIR="$cuSt"
+    CONFIG_JSON="$cuSt/config.json"
+    RIGFORGE_CONTROL_STATE="$cuSt/state"
+    CONTROL_UPGRADE_MIN_INTERVAL=0
+    # Snapshot the status file at the moment the build half runs — the started record must already
+    # be there, and must carry this change's id (not the previous change's terminal record).
+    _control_upgrade_do() {
+        cp "$cuSt/state/status.json" "$cuSt/mid-status.json" 2>/dev/null
+        return 0
+    }
+    _wait_miner_live() { return 0; }
+    git() { echo v0.0.1; }
+    set +e
+    PATH="$STUBS:$PATH" control_upgrade >/dev/null 2>&1
+)
+sMid="$(cat "$cuSt/mid-status.json" 2>/dev/null)"
+assert_eq "started record served while the build runs (#320)" "$(st "$sMid")" "started"
+assert_contains "started record carries this change's id" "$sMid" "abc123def4567890"
+assert_eq "started record indexed under changes/<cid> too (#320)" \
+    "$([ -f "$cuSt/state/changes/abc123def4567890.json" ] && echo y || echo n)" "y"
+assert_eq "terminal record supersedes started" "$(st "$(cat "$cuSt/state/status.json" 2>/dev/null)")" "applied"
 
 # #308: the control-upgrade oneshot runs as root with NO $HOME, so git can't read root's safe.directory
 # config and fatals on "dubious ownership" of the operator-owned install — every git op then fails and

util/control-server.py

@@ -16,7 +16,9 @@
   - Body size is capped; the staged file is written atomically (temp + rename + fsync) so the
     path unit never sees a half-written request.
 
-Python3 stdlib only. GET /status returns the last applied change's result; POST /apply stages one.
+Python3 stdlib only. GET /status returns the last recorded change record — a terminal outcome, or
+control-upgrade's non-terminal `started` while the oneshot runs (#320). POST /apply stages a config
+change; POST /upgrade stages a release upgrade (#308, gated by control_upgrade).
 """
 import hmac
 import json
@@ -227,7 +229,7 @@ def _handle_upgrade(self):
         except OSError as e:
             return self._send(500, "Internal Server Error", {"error": "could not stage upgrade: %s" % e})
         self._send(202, "Accepted", {"status": "accepted", "change_id": cid,
-                                     "note": "queued; the rig refuses anything that isn't a real, reachable release newer than the one installed (it does not independently verify the target is the newest). poll GET /status"})
+                                     "note": "queued; the rig refuses anything that isn't a real, reachable release newer than the one installed (posting the installed version ends 'noop'; it does not independently verify the target is the newest). poll GET /status"})
 
     def _handle_apply(self):
         change = self._read_json_body()