feat: enforce portable experiment execution envelopes

Author: lhy0718

TODO.md

@@ -10,17 +10,36 @@ Usage rules:
 
 ## Now
 
-- Make live-validation temp workspace builders emit `run_status.json` with `validation_scope=live_fixture`.
-  - Current temp workspaces can still look noisier than intended in `/doctor` because older fixtures do not emit the new scope marker.
-  - Keep this additive; do not weaken the default governed-run validator.
-
 - Decide whether to add an additive backfill helper for older runs that predate `run_status.json`.
   - Current behavior is honest fallback projection.
   - Follow-up is whether a bounded backfill would improve operator surfaces without pretending old runs had artifacts they never wrote.
 
-- Add a per-run completeness checklist artifact.
-  - Summarize presence of `run_record.json`, `events.jsonl`, checkpoints, key node artifacts, and public mirrors.
-  - Keep this separate from harness validation and do not turn it into a new gate by default.
+- Validate CPU and explicitly selected NVIDIA GPU isolation on a host that
+  permits user namespaces.
+  - The Local ACI now binds CPU/NVIDIA device policy, mounts only declared
+    numeric GPU IDs plus required NVIDIA control devices, and emits `enforced`
+    only after an in-sandbox start marker.
+  - This workstation rejects the required namespace setup, so its live runs
+    remain honestly `partial` even though the device policy is now implemented.
+
+- Validate the Docker execution adapter against a real purpose-built runtime
+  container on another host.
+  - The adapter now fails closed, clears the command environment, verifies the
+    strict container boundary before and after execution, and checks exact
+    CPU/NVIDIA exposure.
+  - Unit and fake-CLI integration coverage pass locally; a real Docker daemon
+    and selected NVIDIA device run are still required for cross-host evidence.
+  - Remote execution still requires a dedicated enforced adapter.
+
+## Completed Recently
+
+- Added portable execution envelopes, hash-bound receipts, automatic
+  `bubblewrap` isolation, strict inspected Docker execution, explicit GPU
+  device policy, and fail-closed paper-grade execution grounding.
+- Live-validation fixture builders emit `run_status.json` with
+  `validation_scope=live_fixture`.
+- Per-run completeness checklists summarize canonical run artifacts without
+  replacing harness validation or becoming a default gate.
 
 ## Next
 

docs/architecture.md

@@ -397,6 +397,51 @@ manager entry, and `run_experiments` all consume the same recomputed execution
 authorization. Persisted pre-probe booleans and estimator status are diagnostic
 components and must never be treated independently as execution permission.
 
+Every command launched by `run_experiments`, including preflight, primary,
+retry, and supplemental commands, crosses the shared execution-envelope
+boundary. The portable envelope binds the command template and hash, workspace
+relative working and writable roots, an environment-name allowlist, network
+policy and purpose, timeout, declared seeds, explicit CPU/NVIDIA device policy,
+input hashes, dependency-lock hashes, and expected outputs. Per-attempt envelopes and receipts remain under
+`execution/{envelopes,receipts}/`; the latest primary pair is projected as
+`execution/execution_envelope.json` and `execution/execution_receipt.json`.
+
+An ACI adapter must report which controls it actually enforced. The local ACI
+first probes the Linux `bubblewrap` adapter. For local CPU execution, no GPU
+device is exposed. For NVIDIA execution, the requested GPU count and concrete
+numeric visible-device IDs must agree; only those device nodes plus required
+NVIDIA control nodes are mounted. When the host permits namespace creation and
+the device policy is enforceable, the adapter exposes only the
+declared workspace, overlays the explicit writable roots, clears the host
+environment, applies the network policy, and reports `enforcement=enforced`
+only after an in-sandbox start marker is observed. A failed setup does not
+silently rerun the command outside the sandbox.
+
+For `executionProfile=docker`, the ACI uses a dedicated `docker_exec` path
+instead of embedding a second Docker command in the experiment command. Before
+execution it inspects the selected container and requires a stable running
+state, a non-root user, a read-only root filesystem, dropped capabilities,
+`no-new-privileges`, non-privileged mode, identity bind mounts restricted to
+the declared workspace and writable roots, the requested network mode, and an
+exact CPU/NVIDIA device match. The command runs through `/usr/bin/env -i` with
+only the managed allowlist and device visibility variables. A second inspect
+must reproduce the same boundary fingerprint before the receipt can report
+`enforcement=enforced`. An inspect or policy failure blocks the command rather
+than falling back to host execution.
+
+If `bubblewrap` is missing or forbidden by host policy, GPU IDs are absent or
+unsupported, or required device nodes are unavailable, local execution strips
+non-allowlisted environment variables and verifies the command, input hashes,
+and timeout. It does not claim workspace, mount, device, or blocked-network
+isolation and therefore reports `enforcement=partial` with a reason code.
+Remote and plan-only profiles fail closed until they have dedicated adapters.
+An adapter without the envelope API reports
+`enforcement=compatibility`. Neither level is paper-grade. The existing
+`execution_grounding` research-process check requires a hash-valid
+`enforcement=enforced` receipt with every environment, workspace, input,
+timeout, network, mount, and device assurance flag and required output
+present before experimental output can be paper-ready eligible.
+
 `analyze_papers` revalidates the collection generation, query plan, reviewer
 input hash, pair judgments, retained paper IDs, family counts, and semantic
 precision floors. A retired quality artifact, a changed judgment, or a corpus

docs/experiment-quality-bar.md

@@ -64,6 +64,17 @@ When execution is recorded as successful:
 - `run_manifest.json` should exist and summarize what was run.
 - `experiment_portfolio.json` or an equivalent design artifact should exist and describe the planned trial groups.
 - At least one concrete run record must exist under the run artifact tree.
+- `execution/execution_envelope.json` and
+  `execution/execution_receipt.json` must exist, bind each other by SHA-256,
+  and name the actual assurance level. Runtime success may use a `partial` or
+  `compatibility` receipt, but paper-scale promotion may not.
+- `bubblewrap` availability alone is insufficient: an `enforced` receipt
+  requires a successful host probe and an adapter-controlled marker proving
+  that the experiment command started after namespace and mount setup.
+- GPU-backed paper-scale execution must bind a positive requested GPU count and
+  the same number of concrete numeric visible-device IDs. Missing IDs, UUID-only
+  device selectors, unavailable device nodes, and Docker/remote execution through
+  the local adapter remain `partial` until a dedicated enforced adapter exists.
 
 ### B. `analyze_results` success expectations
 When `analyze_results` is completed:
@@ -121,6 +132,12 @@ For an experiment result to count as paper-scale candidate evidence, all of the
 6. The experiment output is connected to the stated research question.
 7. The run is not merely a smoke test of the workflow itself.
 8. The evidence goes beyond a single thin run by including repeated trials/folds/seeds or explicit robustness evidence such as confidence intervals, stability metrics, or effect estimates.
+9. The primary execution has a hash-valid `enforcement=enforced` receipt whose
+   environment, workspace, input-hash, timeout, network-policy, and mount-
+   isolation and device-policy assurances all pass, whose dependency lock is
+   present, and whose required outputs are hash-bound. Docker receipts require
+   matching pre/post container-boundary inspection; an uninspected
+   `docker exec` is not paper-grade execution evidence.
 
 If any of the above is missing, the result may still be valid runtime output,
 but it must not be treated as paper-worthy experimental evidence.

docs/reproducibility.md

@@ -28,6 +28,10 @@ Reproducibility claims must be backed by concrete artifacts.
   or public output
 - Planned portfolio / trial-group structure (`experiment_portfolio.json`)
 - Run manifest (`run_manifest.json`)
+- Portable primary execution envelope and hash-bound receipt
+  (`execution/execution_envelope.json`,
+  `execution/execution_receipt.json`), plus immutable per-attempt records under
+  `execution/envelopes/` and `execution/receipts/`
 - Matrix trial-group index when managed bundle execution materializes dataset/profile slices (`trial_group_matrix.json`)
 - Per-slice managed trial-group metrics when present (`trial_group_metrics/*.json`)
 - Raw or summarized metrics (`metrics.json`, supplemental metrics)
@@ -49,6 +53,38 @@ For each run, preserve:
 - key gate/recovery artifacts (`transition_recommendation.json`, `collect_background_job.json` when present)
 - key generated artifacts in `.autolabos/runs/<run_id>/...`, including trial-group matrix artifacts when present
 
+Execution-envelope reproducibility requires more than the files being present:
+
+- Recompute `envelope_sha256` after removing only `envelope_id` and
+  `envelope_sha256`, and require `envelope_id` to match that digest.
+- Recompute `receipt_sha256` after removing only `receipt_sha256` and require
+  its run, phase, attempt, envelope ID, and envelope hash to match the envelope.
+- Keep command paths portable through `${WORKSPACE_ROOT}` and store artifact
+  paths relative to the workspace. Never persist environment values or secrets.
+- Bind `devices.policy`, `requested_gpu_count`, and concrete visible device IDs
+  in the envelope. CPU-only execution exposes no GPU device. NVIDIA execution
+  is paper-grade eligible only when the declared count and IDs agree and the
+  adapter mounts exactly the selected GPU device nodes plus required control
+  nodes.
+- Treat `partial` and `compatibility` receipts as development diagnostics, not
+  paper-grade execution evidence. A paper-facing execution requires an
+  `enforced` adapter receipt, verified dependency locks, required outputs, and
+  all environment, workspace, input, timeout, network, mount, and device
+  assurances.
+- The local ACI may issue `enforced` only when its `bubblewrap` probe succeeds
+  and the command emits the adapter-controlled in-sandbox start marker. Missing
+  namespace permission, setup failure, or an incomplete GPU device declaration
+  must remain visible as a reason-coded non-paper-grade receipt.
+- The Docker ACI may issue `enforced` only when pre/post
+  `docker container inspect` observations match and both satisfy the envelope: stable non-root
+  and non-privileged execution, read-only root filesystem, dropped
+  capabilities, `no-new-privileges`, bounded identity bind mounts, requested
+  network mode, and exact CPU/NVIDIA device exposure. The process environment
+  must be rebuilt through `env -i`. Inspection or policy failure blocks the
+  command; it must not fall back to host execution.
+- Remote and plan-only profiles remain fail-closed until a dedicated adapter
+  can emit equivalent independently checkable assurances.
+
 ## 3) Model-review reproducibility
 
 Model-assisted review is reproducible only when its inputs, topology, and

src/core/evaluation/evalHarness.ts

@@ -338,6 +338,7 @@ async function evaluateRun(runsDir: string, run: RunRecord): Promise<EvalHarness
   const policyBlocked = implementFailureType === "policy" || runVerifierStage === "policy";
   const storedStatus = await readRunOperatorStatus(runDir);
   const researchProcess = storedStatus?.research_process || await buildRunResearchProcessProjection({
+    workspaceRoot: path.resolve(runsDir, "..", ".."),
     runDir,
     run
   });