fix(mcp): auto-recover remote MCP OAuth on server-side invalid_token

[aheritier]

---

Summary

Fixes the bug where a remote MCP server rejecting a stored OAuth access token with a spec-compliant 401 invalid_token was not auto-recovered: the lifecycle supervisor treated the auth failure as a generic transient error, burned all 5 reconnect attempts, and the toolset instead permanently in StateFailed. The only workaround was restarting the process. Observed with the Notion remote MCP server (https://mcp.notion.com/mcp) but the root cause is transport-agnostic.

Fixes #3198

Root causes addressed

1. Error misclassification (pkg/tools/lifecycle/classify.go): a 401/invalid_token was not mapped to the existing ErrAuthRequired sentinel, so IsPermanent returned false and the supervisor retried 5× pointlessly before failing.
2. Silent refresh only on local expiry (pkg/tools/mcp/oauth.go): refresh fired only when the token was locally expired; a server-side rejection of a token we still believed valid was never refreshed or evicted.
3. Re-auth handler never ran on the reconnect path (pkg/tools/mcp/oauth.go + pkg/tools/lifecycle/supervisor.go): background reconnects had no ready elicitation bridge, so the toolset silently died with no re-auth affordance.

What changed

Commit 1 — fix(mcp): stop retry storm on permanent OAuth reconnect error

• classifyByMessage maps invalid_token / invalid token (RFC 6750 §3.1) to ErrAuthRequired. A bare unauthorized without invalid_token is intentionally NOT matched, to avoid false positives on app-level 401s.
• supervisor.tryRestart now checks IsPermanent after connector.Connect fails and transitions straight to StateFailed instead of exhausting the retry budget (symmetric with the existing Wait()-path check).
• Adds withBackgroundReconnect/IsBackgroundReconnect context helpers; clientConnector.Connect applies WithoutInteractivePrompts for background reconnects so a background 401 defers cleanly, and maps auth/deferred-OAuth errors to lifecycle.ErrAuthRequired.

Commit 2 — fix(mcp): evict + refresh on server-side OAuth invalid_token rejection

• errorCodeFromWWWAuth extracts the RFC 6750 error= code from the WWW-Authenticate header.
• refreshStoredToken is extracted from getValidToken and shared by the local-expiry and server-rejection paths; it honours the existing 30s refreshFailedAt backoff.
• roundTrip tracks whether a token was attached; on a 401 with invalid_token signalled and a token attached, it runs handleServerRejectedToken (serialised under oauthFlowMu so concurrent initialize-stage RPCs coalesce): re-check, evict stale token, attempt silent refresh, otherwise fall back to interactive OAuth or defer. The lastOAuthDeclined sticky-decline latch is honoured so a dismissed dialog is not re-popped.

Commit 3 — fix(mcp): surface re-auth notice and ensure background reconnects are non-interactive (Fixes #3198)

• StartableToolSet gains a recoveryStreak (distinct from startStreak) and ShouldReportRecoveryFailure(), which fires exactly once per recovery-failure streak and stays silent for initial-startup auth deferral.
• emitToolsProgressively (and agent.ensureToolSetsAreStarted) emit a single deduped, user-visible notice — " needs re-authentication — it will prompt on your next message, or use /toolset-restart" — when a previously-working toolset fails with ErrAuthRequired.

Commit 4 — docs: document invalid_token auto-recovery behavior

• Documents the silent-refresh path and the re-auth-on-next-message fallback in docs/features/remote-mcp, docs/community/troubleshooting, and docs/tools/mcp.

Behaviour after this change

• A server-side invalid_token no longer triggers a 5-retry storm.
• When a refresh token is available, the toolset silently refreshes and reconnects — fully automatic, no user interaction.
• When refresh is not possible, the user gets a clear one-time re-auth notice and the OAuth prompt reappears on the next message (or via /toolset-restart).

Testing

• task build, task lint (clean for all changed files), and go test -race ./pkg/tools/lifecycle/... ./pkg/tools/mcp/... ./pkg/tools/... ./pkg/runtime/... ./pkg/agent/... all pass.
• New/updated tests:
  • TestSupervisor_PermanentConnectErrorDoesNotRetry — permanent Connect error fails fast (2 calls, not 5).
  • classify_test.go — invalid_token → ErrAuthRequired, including idempotency and a bare-unauthorized negative case.
  • TestRoundTrip_ServerInvalidTokenEvictsAndRefreshes, ..._RefreshFails_DefersWhenNonInteractive, ..._NoRefreshToken_NonInteractive, TestRoundTrip_FirstContact401Unchanged, TestRoundTrip_ConcurrentInvalidToken_RefreshesOnce, TestRoundTrip_Bare401WithAttachedTokenUnchanged, TestRoundTrip_ConcurrentInvalidToken_NoRefresh_StickyDecline.
  • TestEmitStartupInfo_RecoveryAuthNoticeEmittedOnce and the StartableToolSet.ShouldReportRecoveryFailure suite (once-per-streak, not-on-initial-startup, resets on success/stop).

Notes

• Commits are SSH-signed.
• No config/schema changes.