watchdog: auto-recover a wedged miner + optional thermal cutoff (timer, opt-in)

[VijitSingh97]

Why

Restart=always in the xmrig unit only covers a dead process. The failure mode it can't see: xmrig wedges — process alive, 0 H/s, HTTP API unresponsive — and systemd happily reports active (running) while the rig mines nothing until a human notices. On a fleet that runs 24/7 unattended, "notices" can be days.

Second unattended gap: sustained thermal stress. #62 (closed) made tune-time measurements thermal-aware — reject throttled readings while picking a config; it deliberately did not touch steady-state runtime protection. A failed case fan in July shouldn't cook a rig because nothing watches temperature between tune runs.

Both checks are the same shape as the existing periodic-autotune machinery (#46/#95): a systemd timer firing a oneshot that runs a rigforge verb. Copy that pattern; invent nothing.

Scope

• New verb watchdog: one health check per invocation (the timer provides cadence). Logic:
  1. Service not active → reset state, exit 0 (systemd Restart= owns dead-process recovery; we own wedged-process recovery).
  2. Thermal hold active (marker file exists) → read temp; below max_temp_c - 5 → clear marker, start the service, log; else exit (stay stopped).
  3. max_temp_c set and temp above it → stop the service, write the marker, log, exit.
  4. Local API (127.0.0.1:8080/2/summary) unreachable or hashrate 0 → bump a consecutive-fail counter (state file in $WORKER_ROOT); at 2 consecutive fails → systemctl restart, reset counter, log. Healthy → reset counter.
• rigforge-watchdog.timer + oneshot .service templates, installed/removed by a new install_watchdog() copied from install_autotune() (rigforge.sh:772-796), wired into main() (after line 996), apply() (rigforge.sh:2700-2706 — apply is the config-change path), and uninstall() (next to the autotune-timer removal, rigforge.sh:1106-1109).
• Config keys (reference file only; config.minimal.json stays minimal): watchdog = "disabled" (default) | "enabled"; watchdog_interval_min = 5; max_temp_c = "" (empty = no thermal cutoff).
• Linux-only, like install_autotune ([ "$OS_TYPE" == "Linux" ] || return 0) — macOS has no systemd and is the dev/light-use platform.

Out of scope: notifications/alerting (a future issue can tail the journal), per-pool health, temperature tuning (that's #62's domain — this only stops/starts), configurable hysteresis (fixed 5°C — one less knob; revisit only if a real rig oscillates).

Acceptance criteria

• "watchdog": "enabled" + setup/apply installs rigforge-watchdog.timer firing every watchdog_interval_min minutes; "disabled" (or absent) removes both units cleanly; uninstall removes them.
• Two consecutive checks seeing hashrate 0 / API unreachable restart the miner and say so in the journal; one blip does not.
• With max_temp_c set, temp above it stops the miner and leaves a marker; a later run below max_temp_c - 5 starts it again exactly once.
• max_temp_c empty/absent = thermal logic fully skipped.
• ACCESS_TOKEN rigs work (the check sends the Bearer header) and the token never appears in logs or unit files.
• Invalid values hard-error in parse_config (typo ≠ silently disabled), matching the autotune tri-state precedent at rigforge.sh:363-368.
• Tests cover: install/remove toggle, restart-after-2-fails, single-fail no-restart, thermal stop + hysteresis restart, disabled thermal path. make lint + make test pass (macOS bash 3.2 and Linux).
• Documented in config.reference.json, docs/configuration.md, docs/operations.md; usage() gets the verb.

Implementation notes

Numbered, mechanical. All design decisions are made — do not reopen them.

1. parse_config (after rigforge.sh:374): parse the three keys.
   • watchdog: copy the autotune case-normalization shape (rigforge.sh:363-368): disabled|false|off|none|null|"" → WATCHDOG_MODE=disabled, enabled|true|on → enabled, *) error (typo must not silently disable a recovery mechanism).
   • WATCHDOG_INTERVAL_MIN=$(jq -r '.watchdog_interval_min // 5' ...); validate integer 1-1440 with the [[ =~ ^[0-9]+$ ]] pattern used for DONATION (rigforge.sh:288).
   • MAX_TEMP_C=$(jq -r '.max_temp_c // empty' ...); empty allowed; else integer 40-110 (below 40 you'd never restart; above 110 is meaningless). Default empty because a wrong cutoff on unknown sensors (thermal_zone0 varies by board) is worse than no cutoff — it's opt-in for operators who've checked their zone.
2. Templates in systemd/, copied from the autotune pair:
   • rigforge-watchdog.service.template: [Unit] Description=RigForge miner watchdog check, After=$SERVICE_NAME.service; [Service] Type=oneshot, Environment=RIGFORGE_OPERATOR=$RIGFORGE_OPERATOR (same re-own rationale as the autotune unit), ExecStart=$SCRIPT_DIR/rigforge.sh watchdog. Do NOT bake ACCESS_TOKEN or max_temp_c into the unit — the verb re-reads config.json via parse_config, so apply after a config edit needs no unit rewrite (only the interval lives in the timer).
   • rigforge-watchdog.timer.template: [Timer] OnBootSec=${WATCHDOG_INTERVAL_MIN}min, OnUnitActiveSec=${WATCHDOG_INTERVAL_MIN}min, [Install] WantedBy=timers.target. Interval semantics ("every N minutes"), so OnUnitActiveSec not OnCalendar — the autotune timer's calendar cadence doesn't fit minutes.
3. install_watchdog() directly below install_autotune() (after rigforge.sh:796): literal copy of its shape — Linux guard, disabled → systemctl disable --now + rm -f both units + daemon-reload (idempotent, only if present), enabled → envsubst-render both templates ('$SERVICE_NAME $RIGFORGE_OPERATOR $SCRIPT_DIR' for the service, '$WATCHDOG_INTERVAL_MIN' for the timer) via sudo tee, daemon-reload, enable --now.
4. Wiring: main() — add CURRENT_STEP="configuring the watchdog"; install_watchdog after the install_autotune call (rigforge.sh:995-996). apply() — add install_watchdog >/dev/null 2>&1 || true beside install_autotune (rigforge.sh:2703). uninstall() — add the watchdog pair to step 1 next to the autotune-timer block (rigforge.sh:1106-1109). Dispatcher (bottom case, rigforge.sh:~3087): watchdog) watchdog ;;. usage(): one line under "Day to day" or "Tuning" ("watchdog — one health check: restart a wedged miner, enforce max_temp_c; scheduled via watchdog:"enabled"").
5. watchdog() verb (place near apply()/bench()): parse_config first (gets WORKER_ROOT, ACCESS_TOKEN, MAX_TEMP_C). State: $WORKER_ROOT/watchdog.fails holding a bare integer (plain file, not JSON — bash 3.2, no parsing needed), thermal marker $WORKER_ROOT/watchdog.thermal-hold. Steps:
   • systemctl is-active --quiet "$SERVICE_NAME" false and no thermal marker → rm -f the fails file, log, return 0.
   • Temp: reuse _read_temp (rigforge.sh:1290-1299) verbatim — it already handles the thermal-zone read, THERMAL_ZONE override, and the TUNE_TEMP_CMD test override. TUNE_TEMP_CMD stays env-only; never read a command string from config.json and eval it (config is operator-writable without root; eval'ing it is privilege escalation).
   • Thermal hold: marker exists → if MAX_TEMP_C set and temp non-empty and awk says temp < MAX_TEMP_C - 5 → rm -f marker, systemctl start, log "watchdog: temp ${t}C below $((MAX_TEMP_C-5))C — restarting the miner."; else return 0. Hysteresis fixed at 5°C: big enough to outlast the post-restart heat-up on every rig we run (7800X3D/EPYC), small enough to not strand the miner; not configurable (one less knob to typo).
   • Cutoff: no marker, MAX_TEMP_C set, temp non-empty, temp > MAX_TEMP_C (awk — temps are floats) → systemctl stop, touch marker, warn, return 0. Unreadable temp = skip thermal (a missing sensor must not stop a healthy miner).
   • Hashrate: reuse _read_api_hashrate (rigforge.sh:2328-2343) verbatim — it already handles the ACCESS_TOKEN Bearer branch, the no-token default, and the API_CMD test override. Empty (unreachable) or awk-zero → increment the fails file (f=$(cat ... 2>/dev/null || echo 0)); f >= 2 → systemctl restart, reset file to 0, warn "watchdog: miner wedged (0 H/s / API dead twice) — restarted."; else write the bumped count and log. Healthy → reset to 0. Two consecutive fails, decided: one 5-min sample can be a restart-in-progress or dataset re-init; two misses = ~10 min wedged, unambiguous. Logging = the existing log/warn helpers; a oneshot's stdout/stderr lands in the journal — no logger needed. Never print ACCESS_TOKEN.
6. Config/docs: config.reference.json — add "watchdog": "disabled", "watchdog_interval_min": 5, "max_temp_c": "" next to autotune, and extend the _docs sentence. config.minimal.json untouched. docs/configuration.md + docs/operations.md: a short "Watchdog" section (what it checks, the 2-strike rule, the 5°C hysteresis, that max_temp_c needs a sane thermal_zone0). Coordinate with the config-key-lint issue (filed alongside): whichever lands second adds the three keys to parse_config's allowlist.
7. Tests (tests/run.sh):
   • envsubst stub (tests/run.sh:143-149) must learn $WATCHDOG_INTERVAL_MIN — add one -e "s|\$WATCHDOG_INTERVAL_MIN|${WATCHDOG_INTERVAL_MIN:-}|g" clause, or every rendered timer keeps the literal $WATCHDOG_INTERVAL_MIN.
   • Install/remove toggle: copy the install_autotune block at tests/run.sh:1228-1262 wholesale (mktemp sandbox, copy both templates, subshell that sources $SCRIPT, sets OS_TYPE=Linux SCRIPT_DIR SYSTEMD_DIR REAL_USER WATCHDOG_MODE WATCHDOG_INTERVAL_MIN, runs install_watchdog with PATH="$STUBS:$PATH"); assert both units appear, the timer contains OnUnitActiveSec=7min for an override and 5min for the default, the service bakes RIGFORGE_OPERATOR and invokes rigforge.sh watchdog, and disabled removes both.
   • Check logic: unit-test watchdog() in a subshell with API_CMD='echo "{\"hashrate\":{\"total\":[0]}}"' (and API_CMD='false' for unreachable), TUNE_TEMP_CMD='echo 92', a temp WORKER_ROOT, and the systemctl stub; assert via CALL_LOG grep (the stubs already append every call, tests/run.sh:~155) that restart fires only on the second failed run, that stop + marker happen above max_temp_c, and that start happens once below max_temp_c - 5. Note systemctl is-active via the stub exits 0 (= active) — exactly what the wedged-miner scenario needs.
8. Verification: make lint && make test locally; on miner-0 (release e2e rig): enable the watchdog, kill -STOP the xmrig main thread's hashing (or firewall :8080) and watch two timer ticks restart it; set max_temp_c just below the current temp and watch the stop/marker/restart cycle; journalctl -u rigforge-watchdog.service shows every decision.

Pitfalls: bash 3.2 — no associative arrays, no ${var,,}; set -Eeuo pipefail — every cat/curl/systemctl that may fail needs || true/2>/dev/null handling like _read_api_hashrate does; temps are floats → compare with awk, not [ -gt ]; jq --arg for any JSON writes; never echo ACCESS_TOKEN; the fails/marker files live in $WORKER_ROOT which _reown_worker (rigforge.sh:960-970) re-owns — root-written state there is already handled; make lint = pinned shellcheck 0.11 + shfmt -i 4.

Related

• Tuning: adaptive, benchmark-driven auto-tuning of the XMRig config #46 / autotune: let the periodic run target efficiency (H/s-per-watt), not just raw hashrate #95 — the periodic-autotune timer pattern this copies (install_autotune, rigforge.sh:772-796).
• tune: measure sustained / thermal steady-state (and reject throttled readings) #62 (closed) — tune-time thermal steady-state/throttle detection; this is the runtime protection complement, no overlap: tune: measure sustained / thermal steady-state (and reject throttled readings) #62 picks configs, this stops/starts the service.
• tune: confirm the applied config beats the previous one on the live rig (A/B) #64 (closed) — post-tune live A/B confirm; same "trust but verify the live rig" philosophy applied to health instead of speed.
• Honor & document the Pithead worker-API contract (port 8080 · token = rig name · restricted=true) #24 / Pithead integration — the :8080 API contract the health probe reuses (_read_api_hashrate, rigforge.sh:2328-2343).