Skip to content

Grade benchmarks with runner configs from the pristine exercise tree - #5688

Open
AUTHENSOR wants to merge 1 commit into
Aider-AI:mainfrom
AUTHENSOR:fix/grade-with-pristine-runner-configs
Open

Grade benchmarks with runner configs from the pristine exercise tree#5688
AUTHENSOR wants to merge 1 commit into
Aider-AI:mainfrom
AUTHENSOR:fix/grade-with-pristine-runner-configs

Conversation

@AUTHENSOR

Copy link
Copy Markdown

Summary

run_unit_tests restores only the test files before grading, while the test command runs in the model-writable working tree and the benchmark harness auto-approves file creation (InputOutput(pretty=False, yes=True)). A four-line model-created conftest.py that resets session.exitstatus therefore makes pytest exit 0 while every test fails, and the exercise is recorded as solved:

def pytest_sessionfinish(session, exitstatus):
    if exitstatus != 0:
        session.exitstatus = 0

The chat history still contains the FAILED test lines while the exercise is scored solved, and a gamed run is indistinguishable from an honest one in the results yaml and on the leaderboard, which accepts community-submitted results. The same primitive reaches the other languages through their runner config files (package.json for JavaScript, CMakeLists.txt for C++, build.gradle for Java, Cargo.toml for Rust), none of which are restored.

Fixes #5681.

Changes

  • Before grading, the runner config files for the exercise's language are restored from the original exercise copy when it has them, and any model-created ones are removed when it does not (conftest.py, pytest.ini, pyproject.toml, setup.cfg, tox.ini for Python, plus the equivalents for JS, C++, Java, Rust, Go).
  • As a safety net, a 0 exit code is distrusted when the command's own summary reports failed tests (pytest's "N failed", mocha's "N failing", cargo's "test result: FAILED", go's "FAIL", gradle's "> Task FAILED" / "BUILD FAILED").

Testing

New tests in tests/test_benchmark_run_unit_tests.py:

  • a model-created conftest.py can no longer force a pass: the exercise scores failed, and the conftest is removed before grading
  • a pristine conftest.py is restored over a model-modified one
  • an honest pass still passes
  • the summary-pattern helper matches each runner's failure line and does not match zero-count summaries

Verified behaviorally in both directions with the exact scenario above: on the pre-fix code the exercise with a failing suite is recorded as solved via the conftest; with the change the conftest is removed and the exercise scores failed.

run_unit_tests restored only the test files before grading, while the
test command runs in the model-writable working tree and the harness
auto-approves file creation. A four-line model-created conftest.py
that resets session.exitstatus therefore makes pytest exit 0 while
every test fails, and the exercise is recorded as solved. The same
primitive reaches the other languages through their runner config
files (package.json, CMakeLists.txt, build.gradle, Cargo.toml).

Restore the runner config files from the original exercise copy when
it has them, remove any model-created ones when it does not, and as a
safety net distrust a 0 exit code when the command's own summary
reports failed tests.
@CLAassistant

CLAassistant commented Sep 8, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark scores an exercise as solved based on a test command exit code that the model's own files control

2 participants