docs(wiki): add agent-harness automated testing pipeline design - #295
Open
davida-ps wants to merge 1 commit into
Open
docs(wiki): add agent-harness automated testing pipeline design#295davida-ps wants to merge 1 commit into
davida-ps wants to merge 1 commit into
Conversation
- New wiki design doc: two-phase pipeline (pre-merge skill claims verification + post-release install/verify pipeline verification) - Tiered inference: deterministic mock tier (no secrets, every PR) + gated real-Bedrock tier; keyless GitHub OIDC to a scoped IAM role - Generalizes the existing Picoclaw/Hermes sandbox regressions into one data-driven per-agent adapter; deterministic oracles over LLM-judging - Documents credential/abuse model, threat table, and an M0-M5 roadmap gated on a mandatory Picoclaw feasibility spike - Design only; execution deferred. Registered under Operations in INDEX.md Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
MergerReady to Merge MERGE: this is a docs-only wiki addition, and the diff matches the stated goal of adding the agent-harness pipeline design plus index entry. There are no review discussions or concrete blockers to resolve. Commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User description
Opener Type
Summary
Adds a design/RFC wiki page for a two-phase automated agent-harness testing pipeline. When a skill changes, the supported agent runtimes (OpenClaw, NanoClaw, Hermes, Picoclaw) install the skill non-mutatively and verify it does what it claims (pre-merge); after a release, the install + verification pipeline itself is verified end-to-end against the published, signed artifacts (post-release).
This is a design document only — no pipeline code is written. It is the blueprint that assigned agents implement later, milestone by milestone.
Changes Made
wiki/agent-harness-testing.md— new Operations design doc (~500 lines): context, goals/non-goals, mandatory Picoclaw feasibility spike, architecture (harness/layout + data-driven per-agent adapter generalizing the existing Picoclaw/Hermes sandbox regressions), credential/abuse model (keyless OIDC → scoped IAM role, tiered mock/Bedrock inference, two isolation models, threat-model table), the two phase workflows, reused-assets table, M0–M5 roadmap, open questions, and verification.wiki/INDEX.md— registered the page under Operations + dated update note.Key design decisions
StringEqualsto the protected environment + exact workflow ref; permission policy allowsInvokeModelon one model and denies everything else.pull_requestrun, forks reach the Bedrock tier only via maintainerworkflow_dispatch(pr_number, head_sha)(nopull_request_target).42/43),SPEC.mdfinding schemas, and declared-vs-observedpermissions.jsonsubset checks; LLM-as-judge is last-resort, quorum-voted, fail-closed.Related Issues
Type of Change
Testing
Ran the checks that CI runs on
wiki/**changes — all pass:node scripts/test-wiki-sync-export.mjs— 0 failuresnode scripts/build-github-wiki-export.mjs --source wiki --output <tmp>— Home.md + _Sidebar.md present, no nested markdown, new page includedpython3 scripts/i18n/qa_check.py— pass (new English-only page has no translation pairs to gate)python3 scripts/i18n/link_check.py— no broken local markdown linksChecklist
Generated description
Below is a concise technical summary of the changes proposed in this PR:
Document the proposed two-phase automated agent-harness testing pipeline in the Operations wiki, detailing the harness adapters, manifests, and workflows that agents will follow for pre-merge and post-release verification. Explain how the mock/Bedrock tiers, per-skill conformance specs, credentialing model, and governance/verification walksthroughs fit into the staged roadmap.
Modified files (1)
Latest Contributors(2)
Modified files (1)
Latest Contributors(1)