REVIEW ONLY — the verification tooling from #50, small enough for a machine reviewer

[alhermann]

Do not merge. Every change here is already an ancestor of consolidation/all-campaigns (PR #50). This branch exists so a machine reviewer can actually read the code.

Copilot declined #50: 2,943 files, over its 300-file limit. 2,679 of those are tier-2 fixture scripts — bulk that a reviewer cannot usefully read. This is the remaining 69 files: the code that judges everything else, where a bug is most expensive.

Where to look, in order of how much damage a bug does

1. scripts/run_tier2_fixtures.py::_needle_present — decides whether a fixture passed.

It has been wrong twice in one day, in mirror-image ways:

same_name_files=1   was satisfied by  same_name_files=10     (5235 expectations)
masked_max=1.000000 was satisfied by  unmasked_max=1.000000  (11 expectations)

The second included both control expectations of dune/unmasked_ds_flux_covers_whole_boundary, whose own docstring states the mutation drives unmasked_max to exactly the value that then satisfies its masked_max control. If there is a third hole, it is in this function.

2. scripts/build_execution_ledger.py — decides whether a fixture discriminates, i.e. whether it goes red when the bug it tests is put back.

There are two kinds of mutation control (an in-source T2_MUTATE branch, and a declared from/to recipe applied in a staged copy). Running only the first made 354 of 395 4C fixtures run byte-identically twice and be recorded as "proving nothing". A mutated TIMEOUT no longer earns credit — five rows had rested on that. MUTATION_STALE (a recipe whose anchor text no longer matches, so it substitutes nothing) and VACUOUS_BASELINE are verdicts, never passes.

3. scripts/audit_named_input_keys.py — screens every identifier the knowledge names against the backend's own corpus, including inside deck templates, which is where an invented key does real damage.

It ships a --selftest that proves it still flags the known SOUNDSPEED fabrication on a pre-fix tree and stays silent on the corrected one. A gate that cannot demonstrate it detects the thing it was built for is just another unverified claim.

4. docs/CONSOLIDATION.md, final sections — eleven defects found in this machinery, each a check that returned a confident answer while looking at the wrong thing. Worth reading before the code, as a map of the failure modes.

Fair warning

Several gates here are red on purpose and the reasons are in #50's description. A green suite was never the goal. The most useful review would be a third hole in _needle_present, or a way any of these gates can be satisfied without the underlying claim being true.

[Copilot]

Pull request overview

This PR extracts and hardens the “verification tooling” subset from PR #50 by adding regression tests and helper scripts that make the project’s knowledge/fixture gates harder to spoof (e.g., preventing silent substitution, vacuous fixture passes, stale result snapshots, and circular allowlists).

Changes:

• Adds new regression and contract tests around template variant selection, knowledge payload integrity/cleanliness, pitfall indexing/Signal hygiene, and fixture non-vacuity/mutation-proof requirements.
• Introduces/extends auditing utilities (e.g., mutation harness and two-stage template audit) to ensure recorded evidence actually corresponds to real executions.
• Wires key anti-fabrication / integrity tests into the “knowledge-freshness” GitHub Actions workflow so they run in CI.

Reviewed changes

Copilot reviewed 69 out of 69 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_template_variant_selection.py	Adds regression tests to ensure template variant selection matches the request and does not silently substitute.
tests/test_signals_are_not_boilerplate.py	Adds a gate to prevent excessive reuse of identical Signal: strings within a backend’s pitfalls corpus.
tests/test_signal_verification.py	Tightens tier-2 snapshot flooring by adding a fixture-set fingerprint currency check and updating expected counts/provenance docs.
tests/test_signal_allowlist_is_not_self_certifying.py	Adds a guard preventing per-backend Signal allowlists from “declaring” foreign-library diagnostics as real without linkage evidence.
tests/test_shared_helpers_are_stageable.py	Adds a gate ensuring shared _ helpers in tier-2 fixtures are stageable (directories, not bare files).
tests/test_results_file_is_not_stale.py	Adds checks ensuring tier2_results.json covers enough of the on-disk fixture tree and has non-colliding keys.
tests/test_quoted_diagnostics_are_real.py	Adds/updates a conservative gate verifying quoted diagnostics exist in searchable backend sources (or report UNKNOWN).
tests/test_pitfall_signal_coverage.py	Updates the SPARTA Signal coverage floor and documents corrected counting/provenance.
tests/test_pitfall_format_contract.py	Adds a format contract gate enforcing [Category] + Signal: presence, canonical categories, and no duplicate entries.
tests/test_payload_stays_parseable.py	Adds tests ensuring served knowledge payloads remain valid JSON even when size-capped.
tests/test_no_undisclosed_surrogates.py	Adds a guard preventing templates from claiming to use a backend while actually using a surrogate stack (unless disclosed).
tests/test_no_foreign_solver_diagnostics.py	Adds a gate preventing backends from quoting foreign-solver diagnostics unless clearly retracted or legitimately used.
tests/test_named_input_keys_exist.py	Adds a baseline/ratchet gate that named input keys must exist in the backend corpus, with skip semantics for partial corpora.
tests/test_mcp_tool_docstring_dispatch.py	Extends docstring/dispatch drift detection to handle alias sets (membership tests) in dispatch chains.
tests/test_mcp_stdio.py	Clarifies critic_approved=True usage in smoke tests as parameter exercise (not a verification claim).
tests/test_mcp_instructions.py	Strengthens instructions tests: critic block can’t be disabled, must explain satisfaction path, and gated tools must resolve critic state.
tests/test_kratos_curved_mms.py	Removes pinned measured numbers from docstring; keeps description of the live external gate without leaking results.
tests/test_knowledge_payload_valid_json.py	Adds end-to-end tests ensuring fitted knowledge blocks are valid JSON and preserve load-bearing keys, with announced removals.
tests/test_knowledge_is_discoverable.py	Adds gates ensuring knowledge areas with pitfalls are discoverable/enumerated and reference-only labeling is accurate and delivered.
tests/test_gate_cannot_see_answers.py	Adds structural guard ensuring verification gate modules/prompts cannot see or import exact solutions/answer keys.
tests/test_fourc_thermo_transient_mms.py	Removes pinned measured values from docstrings; keeps structural/gen-only guarantees and describes live gate qualitatively.
tests/test_fourc_inline_tsi.py	Updates documentation/comments to avoid pinning measured values while preserving rationale/structure checks.
tests/test_fourc_audit_corrections_2026_08_03.py	Adds regression tests pinning specific 4C knowledge corrections from an adversarial execution re-audit.
tests/test_fixtures_do_not_assert_on_wall_clock.py	Adds a gate preventing fixtures from deciding pass/fail based on wall-clock timing markers.
tests/test_fixtures_carry_a_mutation_control.py	Adds a gate requiring each fixture to carry a mutation control (either _mutation spec or in-source T2_MUTATE).
tests/test_fixtures_cannot_pass_vacuously.py	Adds a gate preventing fixtures from passing when they’re actually skipping due to missing backends/binaries.
tests/test_febio_elasticity_mms.py	Updates docstring to clarify offline-only assertions and avoid pinning measured convergence numbers.
tests/test_fabrication_gate.py	Adds regression tests for anti-fabrication checks (mesh sanity, residual checks, selection rules, probe coverage).
tests/test_expectations_assert_values.py	Adds a ratcheting gate preventing fixtures whose expectations assert only key= prefixes (value-blind expectations).
tests/test_dune_tier2_claim_coverage.py	Adds a coverage floor check ensuring DUNE pitfall claims have executed fixture coverage mapped via covers.
tests/test_dealii_tier2_claim_coverage.py	Adds a similar coverage floor check for deal.II, including tracked-fixture accounting and mutation-control debt pinning.
tests/test_dealii_poisson3d_mixed.py	Updates docstring to avoid pinning measured values while keeping generator contract guarantees.
tests/test_coverage_floor.py	Updates the fourc coverage floor count and documents the reason (+particle_dem).
tests/test_coupling.py	Adjusts coupling knowledge assertions to assert on contract tokens rather than legacy prose words.
tests/test_coupling_participants_run.py	Adds execution tests that run shipped coupling participant scripts and verify they produce usable exports and cross-code consistency.
tests/test_coupling_pair_fourc_kratos.py	Declares SIDES_COVERED metadata for coupling sides table backing by live pytest runs.
tests/test_coupling_pair_dealii_ngsolve.py	Declares SIDES_COVERED metadata for coupling sides table backing by live pytest runs.
tests/test_coupling_driver.py	Adds tests for stochastic coupling noise-floor measurement and correct reporting channels.
tests/test_coupling_data_staging.py	Updates docstring/comments and keeps regression coverage for coupled-path data staging.
tests/test_catalog_consistency.py	Adds explicit handling for intentionally-absent DUNE API paths and asserts the exemption list remains truly absent.
tests/test_backend_setup.py	Makes backend install-route coverage derive from the registry (avoids hardcoded backend lists) and makes legacy-config test hermetic.
tests/test_attestation.py	Adds regression tests ensuring quantity claims are backed by attestable solver artefacts and tampering is detected.
tests/test_assembled_payload_is_clean.py	Adds a guard scanning the assembled tool output (agent surface) to prevent hard-coded host paths from being served.
scripts/mutate_tier2_fixtures.py	Adds a mutation harness that stages fixtures, applies declared mutations, prechecks vacuity, and asserts mutants kill the fixture.
scripts/audit_two_stage_templates.py	Adds an audit for templates whose “executes” evidence stops short of running the emitted solver script (two-stage templates).
.github/workflows/knowledge-freshness.yml	Wires key anti-fabrication/contract tests into CI’s knowledge-freshness workflow run list.