Add runtime-provider scrutiny synthesis

Author: chai1any

.factory/library/architecture.md

@@ -86,6 +86,7 @@ This mission should extend the existing Rust workspace, not create a parallel sa
 - The facade crate stays thin; runtime ownership lives below it.
 - Stable role behavior belongs on `Agent`/`SandboxAgent`; request- and environment-specific execution belongs on `Runner`, `RunConfig`, and `SandboxRunConfig`, while `RunOptions` is reserved for per-call overrides rather than long-lived sandbox definition.
 - `RunState` remains the canonical public pause/resume boundary, even if sandbox uses dedicated nested session-state types internally.
+- When adding new default hooks or behavior to `ModelProvider`, audit wrapper providers such as `MultiProvider` and forward the new behavior explicitly so routed models preserve provider-specific metadata and request shaping.
 - Runner-created sandbox sessions are runner-managed; caller-injected live sessions remain caller-managed.
 - Sandbox file, shell, and patch operations must stay rooted to the sandbox workspace and must not escape onto the host filesystem.
 - Optional hosted backends must not force third-party provider dependencies into the core runtime crate.

.factory/library/environment.md

@@ -12,6 +12,7 @@ Environment variables, external dependencies, and setup notes.
 - Network access to crates.io for `cargo fetch`, `cargo test` on cold machines, packaging, and publish verification.
 - Docker is currently unavailable in this environment; Docker-backed sandbox validation is blocked until that changes.
 - `rg` is currently unavailable in this environment; `docs/scripts/check_links.sh` depends on it today and must be fixed or shimmed during the mission.
+- Run `.factory/init.sh` before heavy mission work when you need a quick preflight: it performs `cargo fetch --locked` and warns if `rg` or Docker are unavailable in the current environment.
 
 ## Environment Variables
 - `OPENAI_API_KEY`: required for live OpenAI-backed example/integration validation.
@@ -26,3 +27,6 @@ Environment variables, external dependencies, and setup notes.
 - Do not touch or depend on local services already using ports such as `5000`, `7000`, `8790`, or `8791`.
 - Do not store credentials in repo files or commit generated secrets.
 - Do not assume Docker, ripgrep, provider CLIs, audio hardware, or hosted sandbox accounts exist unless the feature explicitly establishes that prerequisite.
+
+## Validator Quirks
+- `cargo test <filter> -- --exact` does **not** match crate-local unit tests by bare test name alone; for `src/lib.rs` unit tests you need the fully qualified module path, or an integration test with a bare exact name if the mission contract expects a short filter.

.factory/services.yaml

@@ -1,4 +1,5 @@
 commands:
+  init: .factory/init.sh
   install: cargo fetch --locked
   typecheck: cargo check --workspace
   build: cargo build --workspace --examples

.factory/validation/runtime-provider/scrutiny/reviews/agent-tools-and-approval-flow-parity.json

@@ -0,0 +1,21 @@
+{
+  "featureId": "agent-tools-and-approval-flow-parity",
+  "reviewedAt": "2026-04-16T16:03:14Z",
+  "commitId": "325191a",
+  "transcriptSkeletonReviewed": true,
+  "diffReviewed": true,
+  "status": "pass",
+  "codeReview": {
+    "summary": "Pass. Commit `325191a` stays within the intended `agents-core` boundary and fixes the feature's exact-match verification gap by adding public-API integration coverage in `crates/agents-core/tests/tool_approval_semantics.rs:1-194`. The underlying approval resume implementation already enforces bound approvals before replaying the pending tool call in `crates/agents-core/src/run.rs:1059-1127`, and the existing facade coverage for nested agent-as-tool state/input preservation remains in `crates/openai-agents/tests/runner_semantics.rs:138-178`. I did not find correctness regressions or missing in-scope contract behavior in the reviewed change.",
+    "issues": []
+  },
+  "sharedStateObservations": [
+    {
+      "area": "knowledge",
+      "observation": "The mission shared state does not currently call out that `cargo test <filter> -- --exact` will not match crate-local unit tests unless the fully qualified module path is used. For parity features that publish short exact-name verification commands, workers may need integration tests or fully qualified test names to make the command execute real tests.",
+      "evidence": "This feature's verification steps in `features.json:697-698` use short exact-name filters for `openai-agents-core-rs`. The handoff for commit `325191a` explicitly states the change was needed so those commands would execute real tests instead of filtering down to zero unit tests, and the worker added `crates/agents-core/tests/tool_approval_semantics.rs` for that purpose."
+    }
+  ],
+  "addressesFailureFrom": null,
+  "summary": "Reviewed the feature entry, handoff, transcript skeleton, commit `325191a`, and the relevant runtime/facade code paths. The feature passes review: approval-gated resume semantics remain correctly bound in core, the facade nested agent-as-tool parity coverage is still present, and the new integration test target makes the milestone's exact verification commands map to real tests."
+}

.factory/validation/runtime-provider/scrutiny/reviews/mcp-discovery-execution-and-safety-parity.json

@@ -0,0 +1,15 @@
+{
+  "featureId": "mcp-discovery-execution-and-safety-parity",
+  "reviewedAt": "2026-04-16T16:02:28.184034Z",
+  "commitId": "502f9bcc0a2170c94346f998f558823fdbb561f2",
+  "transcriptSkeletonReviewed": true,
+  "diffReviewed": true,
+  "status": "pass",
+  "codeReview": {
+    "summary": "The implementation adds local preflight validation for discovered MCP tool arguments in agents-core before remote execution, preserves approval gating and namespace metadata, and the surrounding MCP semantics tests still cover non-streamed execution, reconnect cleanup, resource connection gating, and sanitized failure outputs.",
+    "issues": []
+  },
+  "sharedStateObservations": [],
+  "addressesFailureFrom": null,
+  "summary": "Reviewed the feature handoff, transcript skeleton, commit 502f9bcc0a2170c94346f998f558823fdbb561f2, and the affected MCP code/tests. The feature matches the contract and I did not find blocking or non-blocking correctness issues in the reviewed scope."
+}

.factory/validation/runtime-provider/scrutiny/reviews/openai-hosted-tool-helper-and-replay-parity.json

@@ -0,0 +1,28 @@
+{
+  "featureId": "openai-hosted-tool-helper-and-replay-parity",
+  "reviewedAt": "2026-04-16T16:03:26.083506+00:00",
+  "commitId": "6b6fe4c",
+  "transcriptSkeletonReviewed": true,
+  "diffReviewed": true,
+  "status": "fail",
+  "codeReview": {
+    "summary": "Fail. The commit adds the expected hosted-tool replay and websocket parity regressions, and the public facade still exposes the five hosted helper constructors. But the core change now sanitizes hosted tool artifacts on every `prepare_model_input_items()` call, not just replay/resume input. Because normal run preparation in `crates/agents-core/src/run.rs:557-560` uses this helper, fresh caller history can silently lose completed hosted tool items without matching outputs. That is broader than the feature contract's replay-only sanitation, and it diverges from the upstream split where only resumed/generated history is pruned while caller items are preserved.",
+    "issues": [
+      {
+        "file": "crates/agents-core/src/internal/items.rs",
+        "line": 45,
+        "severity": "blocking",
+        "description": "`compose_replay_input_items()` now unconditionally calls `sanitize_hosted_tool_replay_items(base_items)`, and `prepare_model_input_items()` routes ordinary turn input through this path. As a result, non-resume runs can drop caller-supplied completed hosted tool items such as `shell_call` or `tool_search_call` when their outputs are absent, even though the feature only called for replay sanitation. Upstream keeps normal caller items intact and only prunes resumed/generated history, so this changes live run semantics rather than just fixing replay behavior."
+      }
+    ]
+  },
+  "sharedStateObservations": [
+    {
+      "area": "skills",
+      "observation": "`rust-runtime-worker` currently requires a failing-test-first flow even when part of the assigned parity slice is already implemented and the remaining work is to add exact regression coverage. The worker deviated for that reason, and the handoff's suggested skill update appears justified.",
+      "evidence": "Handoff `skillFeedback.followedProcedure` is false only for step 3 ('Write failing tests first'), with the explanation that nearby websocket behavior was already green and the worker added the missing exact regression coverage plus targeted validation instead. The transcript skeleton also shows the worker otherwise followed the required mission/library-reading procedure."
+    }
+  ],
+  "addressesFailureFrom": null,
+  "summary": "Reviewed the feature handoff, transcript skeleton, commit 6b6fe4c, and the relevant core/openai files. The feature does not pass scrutiny because the replay sanitizer is wired into normal model-input preparation, so it can drop caller-provided hosted tool history outside resume/replay flows."
+}

.factory/validation/runtime-provider/scrutiny/reviews/provider-routing-transport-and-request-metadata.json

@@ -0,0 +1,15 @@
+{
+  "featureId": "provider-routing-transport-and-request-metadata",
+  "reviewedAt": "2026-04-16T16:02:24.966938Z",
+  "commitId": "c8efbc9",
+  "transcriptSkeletonReviewed": true,
+  "diffReviewed": true,
+  "status": "pass",
+  "codeReview": {
+    "summary": "Pass. The feature stays within the intended crate boundaries and adds precise regression coverage for the three assigned parity assertions without uncovering correctness or contract gaps in the underlying routing and OpenAI transport/request-shaping code.",
+    "issues": []
+  },
+  "sharedStateObservations": [],
+  "addressesFailureFrom": null,
+  "summary": "Reviewed handoff, transcript skeleton, commit c8efbc9 diff, and the relevant runtime/provider code paths. The feature correctly validates MultiProvider prefix routing in agents-core and OpenAI Responses transport isolation plus websocket metadata/header/query/conversation forwarding in agents-openai; I did not find blocking or non-blocking issues in scope."
+}

.factory/validation/runtime-provider/scrutiny/reviews/realtime-session-and-transport-parity.json

@@ -0,0 +1,39 @@
+{
+  "featureId": "realtime-session-and-transport-parity",
+  "reviewedAt": "2026-04-16T16:04:56.440710+00:00",
+  "commitId": "8dfb31c26fc9fdc81d5d180833df56c047da869f",
+  "transcriptSkeletonReviewed": true,
+  "diffReviewed": true,
+  "status": "fail",
+  "codeReview": {
+    "summary": "Fail. The commit adds the requested facade regressions and correctly preserves previously-applied non-null realtime settings across partial updates, but the new merge-based implementation still misses an important part of the parity contract: callers cannot explicitly clear previously-applied optional settings, and mixed flat-vs-nested audio updates can leave the public session state internally inconsistent with the payload the model actually sees.",
+    "issues": [
+      {
+        "file": "crates/agents-realtime/src/config.rs",
+        "line": 190,
+        "severity": "blocking",
+        "description": "`RealtimeSessionModelSettings::merge()` and its nested helpers always keep the old value whenever the update field is `None` (for example `update.voice.clone().or_else(|| self.voice.clone())`, `update.speed.or(self.speed)`, and the nested `current.merge(next)` branches at lines 20-168). That means a caller cannot explicitly clear a previously-applied setting such as `voice`, `speed`, `turn_detection`, `transcription`, `noise_reduction`, or `tracing` once it has been set. The feature description says prior settings should be preserved unless explicitly replaced; after this change, replacing a field with 'unset/disabled' is impossible, which is a parity gap versus the Python session-update surface that accepts explicit `None`/null values in its settings dictionaries."
+      },
+      {
+        "file": "crates/agents-realtime/src/session.rs",
+        "line": 289,
+        "severity": "non_blocking",
+        "description": "`update_agent()` stores the merged settings object verbatim in `state.model_settings`, but `session_payload_from_settings()` later resolves audio values by preferring nested `audio.output.*` fields over the legacy flat aliases (`voice`, `speed`, `output_audio_format`) at `crates/agents-realtime/src/openai_realtime.rs:211-222`. If the existing state was configured through the flat aliases and a later partial update uses the nested `audio` shape (or vice versa), `session.model_settings()` can report stale flat alias values even though the payload actually sent to the model uses the newer nested values. Given the docs explicitly support both representations, the public session state should stay normalized instead of exposing contradictory values."
+      }
+    ]
+  },
+  "sharedStateObservations": [
+    {
+      "area": "skills",
+      "observation": "The `rust-runtime-worker` skill still claims workers should add failing tests before implementation, but this worker's transcript shows code edits landed before any of the exact parity tests were executed, while the handoff still reported `followedProcedure: true`.",
+      "evidence": "The transcript skeleton for session `c4810726-24ef-45dd-bac9-7f55150e7099` shows multiple `ApplyPatch` calls before the first `cargo test -p openai-agents-rs --test realtime_semantics ... --exact` executions, yet the handoff JSON marks `skillFeedback.followedProcedure` as true with no deviations."
+    },
+    {
+      "area": "services",
+      "observation": "Workers appear to treat `.factory/init.sh` as a standard preflight step for runtime-provider work, but that command is not captured in `.factory/services.yaml` alongside the other baseline validation/setup commands.",
+      "evidence": "This feature's transcript skeleton includes an early Execute call for `/Users/staticpayload/Mainframe/openai-agents-rust/.factory/init.sh`, while `.factory/services.yaml` only lists install/typecheck/build/test/lint/docs/package commands and no init/preflight entry."
+    }
+  ],
+  "addressesFailureFrom": null,
+  "summary": "Reviewed the feature metadata, handoff, transcript skeleton, skill file, mission guidance, and commit `8dfb31c26fc9fdc81d5d180833df56c047da869f`. The feature does not pass scrutiny because the new merge semantics preserve prior non-null values but still cannot express explicit clearing of realtime settings, and mixed flat/nested audio updates can leave the public session state inconsistent with the applied model payload."
+}

.factory/validation/runtime-provider/scrutiny/reviews/runtime-conversation-tracking-and-session-input.json

@@ -0,0 +1,28 @@
+{
+  "featureId": "runtime-conversation-tracking-and-session-input",
+  "reviewedAt": "2026-04-16T16:03:21Z",
+  "commitId": "f8f58af",
+  "transcriptSkeletonReviewed": true,
+  "diffReviewed": true,
+  "status": "fail",
+  "codeReview": {
+    "summary": "Fail. The commit usefully preserves typed `AgentsError` values for streamed completions (`crates/agents-core/src/internal/streaming.rs`) and adds focused regressions for conversation delta replay, automatic previous-response chaining, session conflict handling, replay limits, and callback-based history rewriting (`crates/agents-core/src/run.rs`). However, the feature is still incomplete for `session_input_callback`: the session preparation path force-restores the original input whenever the callback returns an empty list, so callers cannot intentionally drop replay input and the session store will persist the dropped new-turn items anyway. That still misses the contract's requirement that callback rewrites be deterministic and not corrupt persisted items.",
+    "issues": [
+      {
+        "file": "crates/agents-core/src/internal/session_persistence.rs",
+        "line": 111,
+        "severity": "blocking",
+        "description": "After `session_input_callback` runs, `prepare_input_with_session()` overwrites an empty callback result with `original_input` (`prepared = original_input.clone(); session_input_items = original_input.clone();`). A callback that intentionally returns `[]` therefore cannot suppress replay input, and the supposedly dropped new-turn items are still appended back into the session store. Upstream parity keeps the callback's empty result instead of restoring the original input, so this feature still does not fully satisfy the `session_input_callback` rewrite contract."
+      }
+    ]
+  },
+  "sharedStateObservations": [
+    {
+      "area": "skills",
+      "observation": "The `rust-runtime-worker` skill requires one manual shell verification when public runtime behavior changes, but this feature handoff/transcript only recorded automated Cargo checks and no manual smoke step.",
+      "evidence": "`.factory/skills/rust-runtime-worker/SKILL.md` step 6 says to perform one manual shell verification for user-visible public-facade behavior. The handoff for session `7183a076-d165-4821-a3b0-e3688019b350` lists only `commandsRun`, and the transcript skeleton shows cargo test/check/fmt commands but no temp-crate/example/manual smoke execution."
+    }
+  ],
+  "addressesFailureFrom": null,
+  "summary": "Reviewed the mission artifacts, transcript skeleton, and commit `f8f58af`. The conversation-tracking and streamed-session error work is mostly solid, but the feature should not pass yet because empty `session_input_callback` rewrites are still overridden back to the original input and persisted session items."
+}

.factory/validation/runtime-provider/scrutiny/reviews/runtime-default-runner-and-sync-semantics.json

@@ -0,0 +1,21 @@
+{
+  "featureId": "runtime-default-runner-and-sync-semantics",
+  "reviewedAt": "2026-04-16T16:02:24.033520+00:00",
+  "commitId": "59fc07b",
+  "transcriptSkeletonReviewed": true,
+  "diffReviewed": true,
+  "status": "pass",
+  "codeReview": {
+    "summary": "Pass. The commit correctly aligns the parity verification surface with the already-implemented runtime behavior. `agents_core::run()` and `openai_agents::run()` still resolve through the configurable default runner (`crates/agents-core/src/run.rs:1345-1395` and `crates/openai-agents/src/lib.rs:2-76`), and `Runner::run_sync()` still rejects active Tokio runtimes with a user-facing error (`crates/agents-core/src/run.rs:136-161`). The new integration test in `crates/agents-core/tests/run_sync_semantics.rs:1-16` closes the exact-name gap in the feature's verification command, while the facade test rename in `crates/openai-agents/tests/runner_semantics.rs:66-88` matches the contract wording without regressing behavior.",
+    "issues": []
+  },
+  "sharedStateObservations": [
+    {
+      "area": "knowledge",
+      "observation": "The mission shared state does not document that `cargo test <filter> -- --exact` will not match crate-local unit tests unless the fully qualified module path is used. This feature's verification step used a short exact name, so the worker had to discover the mismatch and add an integration test with a bare test name to make the contract command executable as written.",
+      "evidence": "Feature verification step at features.json:486 is `cargo test -p openai-agents-core-rs run_sync_rejects_active_runtime -- --exact`. The worker transcript shows that after the first targeted run they inspected `cargo test ... --list`, checked the crate layout, and then added `crates/agents-core/tests/run_sync_semantics.rs` so the exact filter would match."
+    }
+  ],
+  "addressesFailureFrom": null,
+  "summary": "Reviewed handoff, transcript skeleton, commit 59fc07b, and the relevant runtime/facade files. The feature passes review: it preserves the intended default-runner and active-runtime semantics while making the milestone's exact verification commands map cleanly onto real targeted tests."
+}