feat: lightweight meta-benchmark harness (unify Environment/verify + metrics; OSWorld/BrowserGym stubs for phase 2)#266
Merged
Conversation
…metrics) Consolidate the three existing openadapt-evals abstractions -- BenchmarkAdapter (adapters/base.py), TaskVerifierRegistry (evaluation/verifier_registry.py), and the flow-side EffectVerifier -- behind ONE runtime_checkable Environment protocol whose verify() folds all three scoring paths into a single call, plus ONE run_meta() runner that emits one JSONL metrics row per (env, task, mode). This is a consolidation over existing code (~80% reuse per the code audit + literature review), not new infra and not an external harness. New openadapt_evals/harness/ package (lazy PEP 562 imports; no openadapt_flow pulled at import time): - protocol.py: Environment protocol + MetaTask (extends openadapt_types BenchmarkTask with demo/verifier/env). - adapters.py: BenchmarkAdapterEnvironment shim (any BenchmarkAdapter -> env, folding evaluate()+registry into verify()); MockMedAdapter + OpenEMRAdapter wrapping the flow Playwright Backend + RestRecordVerifier/FhirEffectVerifier (lazy + injectable/mockable). - runner.py: run_meta() drives record->compile->replay(->heal)->verify and emits MetaMetricsRow (replay_success by the env verifier, structural_rung_rate, model_calls, effect_verdict, wall_ms, cost_usd). Policies wrap existing replay (replay_runner) + hybrid (HybridFlowAgent) + a generic agent driver. - inspect_export.py: portable Inspect eval-log JSON export/round-trip (no inspect_ai runtime dependency). - external.py: OSWorldAdapter + BrowserGymAdapter phase-2 stubs (NotImplementedError + wiring docstrings; nothing installed). Verified with targeted, fully-mocked tests (12 pass): protocol conformance (runtime_checkable), run_meta row correctness (ground truth from verify(), not policy self-report), registry + native effect-verifier delegation, Inspect-log round-trip, and clean NotImplementedError from the external stubs. No live Azure/VM/paid infra exercised. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Phase 1 of a lightweight meta-benchmark, built by consolidating the interfaces that already exist in openadapt-evals — not new infra, not an external harness. Grounded in a code audit + literature review that both concluded "extend openadapt-evals, ~80% reuse."
The repo has three parallel abstractions that each score success a different way:
BenchmarkAdapter(adapters/base.py) —reset/step/evaluate(WAA, mock, local)TaskVerifierRegistry(evaluation/verifier_registry.py) — registered verifiers →VerificationResultEffectVerifier(openadapt-flow) — CONFIRMED/REFUTED/INDETERMINATE against a real system of recordThis PR unifies them behind one
Environmentprotocol whoseverify()folds all three scoring paths into a single call, and onerun_meta()that drives record→compile→replay(→heal)→verify and emits one JSONL row per (env, task, mode).New package
openadapt_evals/harness/(lazy PEP-562 imports; noopenadapt_flowat import time)protocol.py—runtime_checkableEnvironment(reset/observe/act/verify/close) +MetaTask(extendsopenadapt_types.BenchmarkTaskwithdemo/verifier/env).adapters.py—BenchmarkAdapterEnvironmentshim (anyBenchmarkAdapter→Environment, foldingevaluate()+ registry intoverify());MockMedAdapter(flowRestRecordVerifier) +OpenEMRAdapter(flowFhirEffectVerifier) wrapping the flow PlaywrightBackend. Heavy/live collaborators lazy + injectable/mockable.runner.py—run_meta(env, task, policy)→MetaMetricsRow:replay_success(by the env's verifier, not self-report),structural_rung_rate,model_calls,effect_verdict,wall_ms,cost_usd. Policies wrap the existing replay (replay_runner), hybrid (HybridFlowAgent), and a generic agent driver.inspect_export.py— portable Inspect eval-log JSON export + round-trip, without adoptinginspect_aias runtime.external.py—OSWorldAdapter+BrowserGymAdapterphase-2 stubs:NotImplementedError+ docstrings on how each env's native execution verifier wires intoverify(). Nothing installed.Metrics row schema (per env/task/mode)
env, task_id, mode, replay_success, structural_rung_rate, model_calls, effect_verdict, wall_ms, cost_usd(+ diagnostics:reported_success,verifier_score,verifier_source,num_steps,heal_count,structural_rung_counts,error).Verification (targeted, fully mocked — no live Azure/VM/paid infra)
tests/test_harness_meta_benchmark.py— 12 pass: protocol conformance (runtime_checkable),run_metarow correctness (ground truth fromverify(), not the policy's self-report), registry + native effect-verifier delegation, Inspect-log round-trip, cleanNotImplementedErrorfrom the external stubs. Also verified: importing the package does not pull inopenadapt_flow; ruff clean; related existing suites (test_verifier_registry,test_flow_replay_runner) still green.Not in scope (phase 2)
Real OSWorld/BrowserGym wiring (install + native-verifier delegation), and live runs. Do not merge — review only.
🤖 Generated with Claude Code
https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ