feat(auth): per-sandbox authentication to gateway

[TaylorMutch]

Summary

Adds per-sandbox supervisor authentication for gateway RPCs and closes the
cross-sandbox access gap tracked in #1354. Sandbox supervisors now authenticate
as a specific Principal::Sandbox, and gateway handlers authorize access by
comparing that authenticated principal to the sandbox named in each
sandbox-scoped request.

The implementation has two bootstrap paths:

• Docker, Podman, and VM sandboxes receive gateway-minted JWT bootstrap material
  through driver-managed supervisor secret files or guest secret material.
• Kubernetes sandboxes exchange a projected, pod-bound ServiceAccount token for
  the same kind of gateway-minted JWT. The gateway validates the projected token
  with Kubernetes TokenReview, requires the configured sandbox ServiceAccount
  in the sandbox namespace, checks the pod name and UID, fetches the live pod,
  and reads the gateway-owned openshell.io/sandbox-id annotation.

After bootstrap, all drivers converge on the same steady state: the supervisor
presents Authorization: Bearer <gateway-jwt>, refreshes that credential in
memory, and is authorized only for its own sandbox.

Related Issue

Closes #1354

Changes

• Introduces Authenticator/Principal routing for gateway gRPC
  authentication.
• Adds gateway-minted sandbox JWT signing, validation, and refresh support.
• Adds Docker, Podman, and VM bootstrap plumbing that delivers supervisor-only
  JWT files without exposing tokens through public APIs or user entrypoint
  environments.
• Adds Kubernetes ServiceAccount token bootstrap through IssueSandboxToken
  using the Kubernetes TokenReview API.
• Provisions and configures a Helm-managed sandbox ServiceAccount for sandbox
  pods, with support for using an existing ServiceAccount.
• Configures the Kubernetes compute driver with the sandbox ServiceAccount name
  and sets it on sandbox pods while keeping automatic ServiceAccount token
  mounting disabled.
• Restricts Kubernetes bootstrap to the configured sandbox ServiceAccount and
  the configured sandbox namespace.
• Updates the supervisor gRPC client to acquire a bearer credential at startup
  and inject it on every gateway call.
• Enforces per-handler sandbox ID equality for sandbox-scoped RPCs.
• Pins PushSandboxLogs to the first validated sandbox ID in the stream and
  rejects later frames that try to switch sandbox identity.
• Requires persisted sandbox records before IssueSandboxToken or
  RefreshSandboxToken mint a token.
• Adds sandbox debug-rpc helpers for end-to-end authentication testing.
• Mounts sandbox JWT keys in Helm deployments even when local TLS is disabled.
• Updates helm-dev k3d setup to preload the default community sandbox image to
  speed up Kubernetes e2e smoke tests.
• Updates docs, Helm chart tests, and debugging guidance for the new
  per-sandbox identity model.

Implementation Details

Problem Context

Before this PR, sandbox-class handlers trusted a sandbox_id or sandbox name
supplied in the request body. The shared mTLS client certificate only proved
that the caller had a gateway client certificate; it did not prove that the
caller was sandbox A rather than sandbox B. Any holder of that shared credential
could therefore ask for another sandbox's policy, drafts, provider environment,
or related sandbox-private state.

This PR moves the identity decision into the gateway authentication layer. The
router authenticates the caller, inserts a Principal into request extensions,
and handlers compare that principal to the requested sandbox before serving
sandbox-private data.

Shared Gateway Auth Model

The gateway now uses a pluggable authenticator chain. Each authenticator can
produce a Principal, decline so the next authenticator can try, or reject the
request fail-closed.

The steady-state sandbox credential is a gateway-minted Ed25519 JWT. Validation
checks issuer, audience, key ID, expiry, algorithm, and sandbox identity. The
token is intentionally short lived. Refresh mints a replacement for the same
sandbox principal, and older tokens remain valid only until their own expiry.

This JWT is supervisor identity material:

• It is not returned in CreateSandboxResponse.
• It is not stored in public sandbox metadata.
• It is not logged.
• It is kept out of ordinary user entrypoint environments.

Docker, Podman, And VM Bootstrap

Docker, Podman, and VM deployments do not have a platform identity service
equivalent to Kubernetes projected ServiceAccount tokens. For those drivers, the
gateway uses a push-based bootstrap pattern.

At sandbox creation time, the gateway mints a sandbox JWT for the new sandbox
and passes it to the in-process driver boundary as secret material. The driver
writes that token to a supervisor-only file, or VM guest secret material, and
starts the sandbox with OPENSHELL_SANDBOX_TOKEN_FILE pointing at that file.
The supervisor reads the file once at startup and then keeps the active token in
memory.

This path avoids the unsafe parts of the old model:

• The raw token does not cross the public gRPC API.
• The token is not placed in the user command environment.
• The token is scoped to one sandbox ID.
• Refresh rotates the in-memory bearer token without rewriting bootstrap
  material.

Kubernetes Bootstrap

Kubernetes uses a pull-based bootstrap pattern because kubelet can provide a
short-lived, audience-bound, pod-bound ServiceAccount token to the sandbox pod.

The sandbox pod gets a projected ServiceAccount token mounted at a
supervisor-only path. On startup, the supervisor presents that token to
IssueSandboxToken. The gateway validates the token with Kubernetes
TokenReview, verifies the accepted audience, requires the exact configured
sandbox ServiceAccount username, extracts the bound pod name and UID, fetches
the live pod from the sandbox namespace, checks the UID, and reads the
openshell.io/sandbox-id annotation to derive the sandbox identity.

The Helm chart now creates a dedicated sandbox ServiceAccount by default and
passes its name into the gateway's Kubernetes driver configuration. Operators
can disable creation and provide an existing ServiceAccount name. Sandbox pods
continue to set automountServiceAccountToken: false; the only token made
available to the supervisor is the explicit projected token used for bootstrap.

Handler Authorization

Authentication alone is not enough; handlers still need to authorize access to
the requested sandbox.

Direct sandbox_id handlers compare the authenticated
Principal::Sandbox.sandbox_id to the requested ID. Name-keyed handlers resolve
the sandbox name to the canonical ID and then compare. PushSandboxLogs
authorizes the first non-empty batch, verifies the sandbox still exists, stores
that sandbox ID for the stream, and rejects any later batch that names a
different sandbox.

User principals continue through the normal RBAC path. Sandbox principals are
limited to their own sandbox. Anonymous principals are rejected for
sandbox-scoped paths.

Token Lifecycle

IssueSandboxToken is only available to Kubernetes ServiceAccount bootstrap
principals. RefreshSandboxToken is only available to supervisors already
holding a gateway-minted JWT. Both paths require the sandbox record to still
exist before minting a token, so deleted or unknown sandboxes cannot keep
refreshing credentials.

Kubernetes supervisors can recover from restart by repeating the ServiceAccount
bootstrap exchange. Docker, Podman, and VM supervisors use their file or guest
secret bootstrap material and then rely on in-memory refresh for steady state.

Signing Key Persistence

The gateway JWT signing key is persisted through the existing local and Helm
PKI paths. Helm mounts the JWT key material into the gateway even when local TLS
is disabled, because per-sandbox authentication is independent from TLS
enablement.

Design Decisions For Reviewers

• Two bootstrap patterns, one steady-state credential: local and VM drivers push
  supervisor-only bootstrap material; Kubernetes pulls a token through
  ServiceAccount exchange. Both become the same gateway JWT.
• Kubernetes uses TokenReview instead of in-gateway JWT verification so the
  apiserver remains the source of truth for projected ServiceAccount token
  validity and audience acceptance.
• The Helm chart provisions a sandbox ServiceAccount by default rather than
  creating per-sandbox ServiceAccounts in this PR.
• No per-sandbox Kubernetes Secret objects are created for bootstrap.
• No raw token is exposed through public APIs, CreateSandboxResponse, sandbox
  metadata, ordinary user environments, or logs.
• mTLS remains transport protection, not sandbox identity.
• Handler checks are explicit because handlers know which request field
  identifies the target sandbox.

Reviewer Focus Areas

• Docker, Podman, and VM token file handling: supervisor-only placement, no
  leakage into entrypoint environment, and cleanup behavior.
• Kubernetes bootstrap validation: TokenReview, accepted audience, configured
  ServiceAccount, pod name/UID binding, annotation handling, and RBAC scope.
• Handler coverage: every sandbox-private RPC should either call the
  sandbox-scope guard or have a documented reason not to.
• Streaming RPC behavior: PushSandboxLogs should not allow a stream to change
  sandbox identity after validation.
• Signing key persistence: local and Helm deployments must preserve the JWT key
  across gateway restarts; multi-replica gateways must share the same key
  material.
• Token lifecycle edges: unknown or deleted sandbox records must not receive new
  gateway-minted tokens.

Testing

• mise run pre-commit
• cargo test -p openshell-server auth_rpc
• cargo test -p openshell-server auth::k8s_sa
• cargo test -p openshell-server log_stream_scope
• cargo test -p openshell-driver-kubernetes
• mise run helm:docs:check
• mise run helm:lint
• mise run helm:test
• Kubernetes smoke e2e against a fresh k3d cluster via /helm-dev-environment
• Docker smoke e2e
• Podman smoke e2e

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

Label test:e2e applied for f4daea6. Open Branch E2E Checks, find the run for commit f4daea6, and click Re-run all jobs to execute with the label set. The E2E Gate check on this PR will flip green automatically once the run finishes.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-1404.docs.buildwithfern.com/openshell

[github-actions]

Label test:e2e-kubernetes applied for a5cb368. Open the existing run and click Re-run all jobs to execute with the label set. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[dimityrmirchev]

Thanks for addressing my comment with regards to the TokenReview API and also introducing additional security hardening with 2bcf8a9 and 3a15997. The changes look good to me

[johntmyers]

Security/stability review notes from a focused pass on the per-sandbox auth changes.

1. Sandbox callers can drop Authorization and become Principal::User

Refs: crates/openshell-server/src/multiplex.rs, crates/openshell-server/src/auth/authenticator.rs, crates/openshell-server/src/auth/guard.rs

The new router still has paths where a sandbox can avoid Principal::Sandbox by omitting the sandbox JWT. In no-OIDC deployments, the chain falls through to PermissiveUserAuthenticator, producing a synthetic user. In mTLS deployments, missing bearer auth can fall back to the shared peer cert, which is issued as openshell-user. Once represented as Principal::User, the sandbox allowlist and same-sandbox guard no longer constrain the request.

Suggested way forward:

• Treat sandbox-originated transport identity separately from user identity. A request coming from supervisor/shared sandbox mTLS material should not be promotable to Principal::User just because the bearer is missing.
• Consider removing the peer-cert fallback for authenticated gRPC methods once sandbox JWT auth is configured, or only allowing it for known CLI/front-proxy user paths.
• In dev/no-OIDC mode, keep permissive user auth for external user clients, but deny supervisor-callable paths when the request lacks a valid sandbox principal.
• Add regression tests that call a sandbox-allowed method and a user/admin method with no Authorization under no-OIDC and mTLS configurations.

2. VM sandboxes do not receive any supervisor token source

Refs: crates/openshell-driver-vm/src/driver.rs, crates/openshell-sandbox/src/grpc_client.rs

The supervisor now requires OPENSHELL_SANDBOX_TOKEN, OPENSHELL_SANDBOX_TOKEN_FILE, or OPENSHELL_K8S_SA_TOKEN_FILE. The VM driver builds guest environment without any of those values, so VM sandboxes will fail before opening the gateway control stream.

Suggested way forward:

• Plumb DriverSandboxSpec.sandbox_token into the VM guest as a root-only guest file and set OPENSHELL_SANDBOX_TOKEN_FILE.
• Avoid passing the raw token in the process environment if possible.
• Add VM coverage that validates the supervisor reaches ConnectSupervisor after auth is enabled.

3. Existing installs cannot upgrade cleanly when JWT material is missing

Refs: crates/openshell-server/src/certgen.rs

Existing deployments already have server/client TLS material but no JWT secret/files. The new certgen state machine treats that as partial PKI state and aborts, requiring operators to delete all PKI artifacts and rotate TLS just to add JWT keys.

Suggested way forward:

• Split TLS material and JWT material into independently recoverable states.
• If server/client TLS already exist and only JWT material is missing, generate only the JWT keypair/secret.
• Keep the fatal partial-state path for genuinely inconsistent TLS sets, but make “old install missing JWT” a supported migration.
• Add a test for the pre-PR upgrade state: server secret exists, client secret exists, JWT secret missing.

4. A leaked sandbox JWT can be refreshed indefinitely

Refs: crates/openshell-server/src/grpc/auth_rpc.rs, crates/openshell-server/src/auth/sandbox_jwt.rs

RefreshSandboxToken accepts any still-valid gateway JWT and mints another token while the sandbox record exists. There is no jti, token generation counter, revocation state, rotation binding, or proof-of-possession. A one-time bearer leak can therefore become durable sandbox impersonation until the sandbox is deleted.

Suggested way forward:

• Add a jti or token generation claim and persist the currently valid refresh generation per sandbox.
• Rotate on refresh and reject reused/old tokens after a grace window.
• Alternatively, make access tokens non-refreshable and require refresh through a stronger bootstrap proof such as K8s SA token exchange or driver-owned secret material.
• Consider binding refresh to the sandbox transport identity where available.

5. Docker/Podman and debug tooling expose raw gateway JWTs

Refs: crates/openshell-driver-docker/src/lib.rs, crates/openshell-driver-podman/src/container.rs, crates/openshell-sandbox/src/debug_rpc.rs, proto/compute_driver.proto, crates/openshell-core/src/sandbox_env.rs

The proto/core comments say local drivers materialize the token via a per-sandbox file, but Docker/Podman inject the raw JWT into the container environment. That makes the bearer visible through container metadata and privileged exec workflows. openshell-sandbox debug-rpc refresh also prints a fresh token to stdout.

Suggested way forward:

• Use a root-only bind-mounted token file for Docker/Podman and set OPENSHELL_SANDBOX_TOKEN_FILE instead of OPENSHELL_SANDBOX_TOKEN.
• Keep stripping token-related env vars from the user entrypoint, but do not put the credential in container metadata in the first place.
• Gate token-printing debug subcommands behind an explicit dev/test feature or remove raw token output. If a diagnostic is needed, print claims/fingerprint/expiry instead of the bearer.

6. K8s bootstrap trusts a forgeable pod annotation

Refs: crates/openshell-server/src/auth/k8s_sa.rs, crates/openshell-driver-kubernetes/src/driver.rs

TokenReview proves that the caller has a pod-bound token for the shared sandbox ServiceAccount, then the gateway trusts the pod's openshell.io/sandbox-id annotation. Any actor that can create a pod in the sandbox namespace using that ServiceAccount can annotate it with a victim sandbox ID and mint a gateway JWT for that sandbox.

Suggested way forward:

• Verify the pod is OpenShell-owned, not just annotated. For example, check owner references/UID against the corresponding Sandbox CR or a gateway-persisted pod UID recorded at create time.
• Consider per-sandbox ServiceAccounts or a non-forgeable driver-generated nonce stored both in gateway state and the pod spec.
• Add a negative test where a pod with the right ServiceAccount but forged annotation is rejected.

7. Short JWT TTLs break refresh deterministically

Refs: crates/openshell-core/src/config.rs, crates/openshell-sandbox/src/grpc_client.rs

gateway_jwt.ttl_secs accepts any value, but the refresh loop floors its sleep at 60 seconds. With ttl_secs <= 60, refresh happens after expiry. If the gateway is unreachable until after expiry, the loop also keeps trying refresh with an expired gateway JWT rather than re-running bootstrap.

Suggested way forward:

• Enforce a server-side minimum TTL or reject unsupported values at config load.
• Compute refresh delay without a 60s floor when the token lifetime is short.
• On Unauthenticated/expired-token refresh failures, re-run the available bootstrap path where possible, especially K8s SA token exchange.
• Add a TTL=30s regression test and an outage-past-expiry recovery test.