Consolidate twelve campaigns: remove fabricated input keys, and fix the checks that could not tell #165

Workflow file for this run

.github/workflows/knowledge-freshness.yml at f1d4fbf

	name: Knowledge Freshness Check

	on:
	schedule:
	- cron: '0 6 * * 1' # Weekly Monday 6AM UTC
	push:
	paths:
	- 'data/**'
	- 'src/tools/deep_knowledge.py'
	- 'src/backends/*/backend.py'
	pull_request:
	# Broad path filter so the lightweight tests gate every PR that
	# could affect the catalog, the tooling around it, or the ingest
	# pipeline. Pure-docs PRs that touch only README.md / *.md still
	# skip CI (documented intent); anything code-touching runs.
	paths:
	- 'src/**'
	- 'scripts/**'
	- 'tests/**'
	- 'pyproject.toml'
	- '.github/workflows/knowledge-freshness.yml'
	workflow_dispatch: # Manual trigger

	# Both drift jobs auto-file a GitHub issue on the weekly cron when the
	# catalog/fingerprint goes stale. ``issues.create`` needs ``issues:
	# write``; the repo's default GITHUB_TOKEN permission is read-only, so
	# without this block the issue step 403s -- a failure that only shows
	# up the first time real drift is detected (verified by a one-off
	# dispatch run that confirmed the issue is created with this grant in
	# place). ``contents: read`` is all the checkout needs.
	permissions:
	contents: read
	issues: write

	jobs:
	catalog-consistency:
	# Lightweight: greps backend source for the canonical names the catalog
	# promises (currently 4C only; extension pattern in tests/groundtruth/).
	# Always runs (PR + push + cron) -- network-only, no solver installs.
	runs-on: ubuntu-latest
	steps:
	- uses: actions/checkout@v4

	- name: Set up Python
	uses: actions/setup-python@v5
	with:
	python-version: '3.12'

	- name: Install minimal dependencies + introspection backends
	# Install the Python-importable backends that have a probe
	# in tests/groundtruth/ so their catalog-consistency tests
	# actually run in CI rather than skipping. Source-grep
	# backends (4C, deal.II) need no install -- they fetch from
	# raw.githubusercontent.com.
	#
	# When adding a new introspection probe (kratos, dune,
	# fenics) extend this extra list and the test runs
	# automatically on every PR.
	run: pip install -e ".[skfem,ngsolve,dev]"

	- name: Run lightweight gating tests
	# Catalog-consistency + ingest-pipeline tests are the suite of
	# checks that need no solver installs. Keep them together in
	# one fast job that gates every PR. Tee into a file so the
	# issue-creation step can reuse the exact output that produced
	# the failure -- avoiding misleading issues if a second pytest
	# run produces different output (e.g. cache repopulated,
	# network hiccup resolved).
	#
	# THE ANTI-FABRICATION GATES BELONG HERE, and until 2026-08-06 none of
	# them ran anywhere. Eight of them existed and this job named three
	# unrelated tests, so every one of them ran only when a human typed it.
	# That is not a gate, it is a comment: on the day this was noticed the
	# contamination check was failing on 10 of its 18 patterns, with 19
	# leaked sites across 9 files -- including measured convergence orders
	# in served payloads and an exact solution in closed form in the
	# coupling knowledge, which is precisely what the evaluation grades.
	# Nobody had disabled anything; the checks were simply never wired in,
	# and a check nobody runs protects nothing.
	#
	# All of these are pure-Python and read only the tree, so they need no
	# solver install and belong in this fast job:
	# contaminated -- no measured answer / campaign id / exact
	# solution reachable by an agent
	# format_contract -- every pitfall keeps the [Category] tag and
	# Signal: clause that retrieval indexes
	# discoverable -- no knowledge area holds verified content that
	# no surface enumerates
	# fixture_keys -- a fixture's key names a claim that exists, so
	# evidence cannot silently defend nothing
	# assembled_payload -- what the tool actually returns is clean
	# gate_cannot_see_answers
	#
	# test_quoted_diagnostics_are_real is deliberately NOT here: it needs
	# each backend's own source tree to search, which a CI runner does not
	# have, and it would report UNKNOWN for everything. It runs on a
	# machine with the solvers installed.
	run: \|
	set -o pipefail
	pytest tests/test_catalog_consistency.py tests/test_ingest_session.py tests/test_introspect.py \
	tests/test_knowledge_not_contaminated.py \
	tests/test_pitfall_format_contract.py \
	tests/test_knowledge_is_discoverable.py \
	tests/test_fixture_keys_point_at_real_claims.py \
	tests/test_assembled_payload_is_clean.py \
	tests/test_gate_cannot_see_answers.py \
	tests/test_fixtures_carry_a_mutation_control.py \
	tests/test_named_input_keys_exist.py \
	-v -s 2>&1 \| tee /tmp/catalog-test.log
	# test_named_input_keys_exist skips every backend whose corpus is not
	# installed on the runner, and CI has none of the solvers — so here it
	# asserts almost nothing. That is deliberate and it is not sufficient:
	# the gate's real work happens on the machine with the solvers, and the
	# skip messages in this log are the record of how much went unchecked.
	# Read them; a run where everything skipped is not a run where
	# everything passed.
	# Only swallow failures on the weekly cron so the auto-issue step
	# can still run. On PRs and pushes we WANT the workflow to fail
	# so the check is gating.
	continue-on-error: ${{ github.event_name == 'schedule' }}
	id: catalog

	- name: Create issue on catalog drift
	# Only auto-file an issue on scheduled runs. PR failures fail the
	# workflow above and surface through the PR's status check.
	if: steps.catalog.outcome == 'failure' && github.event_name == 'schedule'
	uses: actions/github-script@v7
	with:
	script: \|
	const fs = require('fs');
	const output = fs.existsSync('/tmp/catalog-test.log')
	? fs.readFileSync('/tmp/catalog-test.log', 'utf8')
	: '(test output not captured)';
	await github.rest.issues.create({
	owner: context.repo.owner,
	repo: context.repo.repo,
	title: '⚠️ Catalog/source drift detected',
	body: `Weekly catalog-consistency check failed:\n\n\`\`\`\n${output.slice(-6000)}\n\`\`\`\n\nA backend source change has broken the MCP catalog. Edit the catalog so its parameter keys / DYNAMICTYPE strings / etc. match what the input parser now accepts.`,
	labels: ['knowledge-drift'],
	});

	fingerprint:
	# Only run the heavy ``.[all-solvers,dev]`` install + fingerprint scan
	# on push and on the weekly cron. PRs are gated on
	# ``catalog-consistency`` only -- adding fingerprint to every PR
	# would install all pip-installable solvers (hundreds of MB) on
	# ubuntu-latest and frequently exceed the runner timeout, turning a
	# heavyweight monitoring job into a noisy PR blocker.
	if: github.event_name != 'pull_request'
	runs-on: ubuntu-latest
	steps:
	- uses: actions/checkout@v4

	- name: Set up Python
	uses: actions/setup-python@v5
	with:
	python-version: '3.12'

	- name: Install dependencies
	run: \|
	pip install -e ".[all-solvers,dev]"

	- name: Generate fingerprints
	run: python scripts/fingerprint_solvers.py

	- name: Compare against saved fingerprints
	run: python scripts/fingerprint_solvers.py --compare
	continue-on-error: true
	id: compare

	- name: Create issue on drift
	if: steps.compare.outcome == 'failure'
	uses: actions/github-script@v7
	with:
	script: \|
	const { execSync } = require('child_process');
	const output = execSync('python scripts/fingerprint_solvers.py --compare 2>&1 \|\| true').toString();
	await github.rest.issues.create({
	owner: context.repo.owner,
	repo: context.repo.repo,
	title: '⚠️ Solver API drift detected',
	body: `Weekly fingerprint check found API changes:\n\n\`\`\`\n${output}\n\`\`\`\n\nPlease review and update the knowledge base accordingly.`,
	labels: ['knowledge-drift'],
	});

	- name: Upload fingerprints
	uses: actions/upload-artifact@v4
	with:
	name: solver-fingerprints
	path: data/fingerprints/

Navigation Menu

Uh oh!

Consolidate twelve campaigns: remove fabricated input keys, and fix the checks that could not tell #165

Workflow file

Consolidate twelve campaigns: remove fabricated input keys, and fix the checks that could not tell #165

Uh oh!

Workflow file for this run