Commit 888b5d1
Exclude flaky wall-clock timing tests from default CI
CI was hitting false-positive failures on tests that assert wall-clock
elapsed time, e.g. TestCallawaySantAnnaSEAccuracy.test_timing_performance
failing on PR #328 with "Estimation took 0.103s, expected <0.1s" — a 3%
overshoot that reflects runner noise (BLAS path variation, neighbor VM
contention, cold caches), not a real regression. The existing "20x margin"
comment acknowledged the problem but no fixed threshold can absorb CI
wall-clock variance.
Mark the two CS timing tests @pytest.mark.slow so they're excluded from
default CI (existing addopts already uses `-m 'not slow'`). Tests remain
runnable on-demand via `pytest -m slow` for local benchmarking — same
pattern the TROP suite uses per CLAUDE.md.
Tests marked:
- test_se_accuracy.py::TestCallawaySantAnnaSEAccuracy::test_timing_performance
(method-level marker — rest of the SE-correctness class still runs)
- test_se_accuracy.py::TestPerformanceRegression (class-level marker — all
three parametrized timing cases move out of default CI together)
Left unchanged: test_methodology_honest_did.py::test_m0_short_circuit uses
wall-clock as a proxy for "short-circuit path taken" — that's a correctness
signal, not a performance signal. Marking it slow would remove the check.
Added a TODO.md entry to replace it with a mock/spy in a follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent cd67d60 commit 888b5d1
2 files changed
Lines changed: 16 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
| 101 | + | |
101 | 102 | | |
102 | 103 | | |
103 | 104 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
252 | 252 | | |
253 | 253 | | |
254 | 254 | | |
| 255 | + | |
255 | 256 | | |
256 | 257 | | |
257 | 258 | | |
258 | 259 | | |
259 | 260 | | |
260 | | - | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
261 | 267 | | |
262 | 268 | | |
263 | 269 | | |
| |||
398 | 404 | | |
399 | 405 | | |
400 | 406 | | |
| 407 | + | |
401 | 408 | | |
402 | | - | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
403 | 416 | | |
404 | 417 | | |
405 | 418 | | |
| |||
0 commit comments