Skip to content

Fix #615: OTel tracer leak from test_full_pipeline pollutes test_ping in full-suite runs - #630

Merged
anilmurty merged 3 commits into
Metabuilder-Labs:mainfrom
Axiya3749:fix-issue-615
Jul 28, 2026
Merged

Fix #615: OTel tracer leak from test_full_pipeline pollutes test_ping in full-suite runs#630
anilmurty merged 3 commits into
Metabuilder-Labs:mainfrom
Axiya3749:fix-issue-615

Conversation

@Axiya3749

Copy link
Copy Markdown
Contributor

Fixes test isolation: an earlier test's leaked global state made 4 test_ping.py tests fail only when the full suite runs, training readers to ignore a red suite.

Summary

  • Restore tokenjam.sdk.agent._tracer to its original value in full_stack's teardown instead of leaving it bound to a shut-down provider and closed DB.

Root cause

full_stack (tests/integration/test_full_pipeline.py) rebinds the module-global agent_mod._tracer to a tracer on its own local TracerProvider, then calls provider.shutdown() and db.close() in teardown — but never restores _tracer. Because tests/integration collects before tests/unit alphabetically, this leaks: test_ping.py's record_llm_call() calls then run through the dead TjSpanExporter/closed DuckDB (raising "Connection already closed", logged as "Span export failed" at provider.py:56) instead of the real global provider that test_ping's own proof exporter is attached to. The proof exporter never captures anything, so intercepted is wrongly False and the 4 tests asserting on "intercepted a test span" / interception-dependent output fail.

Fix mirrors the existing correct pattern in tests/agents/test_record_outcome.py: save the original tracer before rebinding, restore it in teardown, before the provider is shut down.

Related issue

Closes #615

Tests / Verification

  • pytest tests/unit/test_ping.py -v — 10/10 pass
  • pytest tests/integration/test_full_pipeline.py -v — 11/11 pass
  • pytest tests/ — full suite: the 4 previously-failing test_ping tests are fixed; confirmed via git-stash isolation that a pre-existing, unrelated failure in tests/unit/test_deadweight.py (a Rich line-wrap issue in _render_deadweight, unrelated to OTel/tracer state) is present with or without this change, both standalone and in the full suite

What's NOT in this PR

  • No change to tokenjam/ source — this is purely a test-isolation fix in tests/integration/test_full_pipeline.py.
  • tests/unit/test_deadweight.py::test_render_deadweight_names_the_dead_server remains failing — confirmed pre-existing and unrelated (fails standalone, independent of this fix and of test ordering). Separate root cause in the CLI renderer; not touched here.

Checklist

  • Tests pass (pytest tests/unit/ tests/synthetic/ tests/agents/ tests/integration/)
  • Lint clean (ruff check tokenjam/)
  • Type check clean (mypy tokenjam/)
  • CLAUDE.md updated (if architecture changed) — N/A, no architecture change
  • Test spans use tests/factories.py (not raw NormalizedSpan) — N/A, no spans added
  • Requested @anilmurty as reviewer (or @-mentioned him above)

@anilmurty

…der-Labs#615)

full_stack (tests/integration/test_full_pipeline.py) reassigns the
module-global tokenjam.sdk.agent._tracer to a tracer bound to its own
local TracerProvider, then shuts that provider down and closes its
DuckDB in teardown, without ever restoring the original _tracer. Since
tests/integration collects before tests/unit, this leaks into every
later test in the session.

test_ping.py's record_llm_call() then routes spans through the dead
TjSpanExporter/closed DB (raising "Connection already closed", logged
as the "Span export failed" warning at provider.py:56) instead of the
global provider that test_ping's own proof exporter is attached to —
so intercepted spans are never actually captured and the 4 ping tests
fail only in full-suite ordering.

tests/agents/test_record_outcome.py already has the correct pattern
for this hazard (save + restore the tracer around a local provider);
full_stack just never applied it. This brings it in line.
@Axiya3749
Axiya3749 requested a review from anilmurty as a code owner July 27, 2026 19:12
@greptile-apps

greptile-apps Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Restores the SDK module’s original tracer during full_stack fixture teardown before shutting down the fixture-local provider and database.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
tests/integration/test_full_pipeline.py Saves the pre-test tracer and restores it during teardown, preventing fixture-local OTel state from leaking into later tests.

Reviews (3): Last reviewed commit: "Merge branch 'main' into fix-issue-615" | Re-trigger Greptile

@anilmurty anilmurty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Axiya3749 — verified the root cause and your fix: the full_stack fixture rebinds the module-global agent_mod._tracer and never restores it, so a later test_ping routes through the dead exporter / closed DuckDB. Reproduced 4 failures on main → 0 with this patch. Clean, correct, mirrors the save/restore pattern already in test_record_outcome.py. Merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_ping fails only in the full-suite run: ordering pollution hides a real isolation bug

2 participants