test: add public-API surface and server recorded tests#2122
Open
Pouyanpi wants to merge 7 commits into
Open
Conversation
github-actions
Bot
added
status: needs triage
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Pouyanpi
force-pushed
the
test/recorded-tests-public-api-surface
branch
from
July 2, 2026 10:05
8b70a2c to
d78c71d
Compare
This was referenced Jul 2, 2026
Pouyanpi
added
status: triaged
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (49)
👮 Files not reviewed due to content moderation or server errors (48)
📝 Walkthrough
🚥 Pre-merge checks | ❌ 1❌ Failed checks (1 inconclusive)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Pouyanpi
added
status: triaged
Contributor
Greptile SummaryAdds a comprehensive recorded characterization test baseline for
|
| Filename | Overview |
|---|---|
| tests/recorded/rails/public_api/test_stream_cancellation.py | Tracks streaming teardown / partial-on-abort / task-cancellation behavior with a blocking async generator; AsyncGenerator annotation is now correctly applied (prior thread resolved). |
| tests/recorded/rails/public_api/test_parity.py | Deterministic parity baseline across all rail pipeline configurations; fully populated inline snapshots, no cassette dependency. |
| tests/recorded/rails/public_api/test_concurrency.py | Validates contextvar isolation under asyncio.gather by re-running the parity matrix concurrently and comparing against a sequential baseline; clean design with intentional cross-module import from test_parity. |
| tests/recorded/rails/public_api/test_generate.py | Provider contract tests for generate/generate_async (OpenAI, NIM) with log stats, options, and error cases; well-structured with cassette-parity assertions gated on record_mode. |
| tests/rails/llm/test_live_smoke.py | Live smoke tests for OpenAI and NIM; NIM test passes options with llm_calls logging enabled but does not assert on the log, leaving that option unused compared to the OpenAI version which validates call recording. |
| tests/recorded/rails/public_api/test_stream.py | Streaming parity matrix (allowed/blocked/boundary/disabled), metadata flag alias, and tool-call streaming; helper _chunks carries an AsyncIterator annotation instead of AsyncGenerator. |
| tests/recorded/server/conftest.py | Server fixture wires TestClient with config-path and rails-instance cleanup; correctly restores rails_config_path in finally, clears llm_rails_instances both before and after. |
| tests/recorded/server/test_chat_completions.py | Server-level contract tests for chat completions, input blocking, and invalid config handling; characterizes current 200-with-error-body behavior per the PR's stated intent. |
| tests/recorded/rails/public_api/test_colang_v2.py | Colang 2.x runtime selection, generate, process_events and self-check-input-rail coverage; xfail placeholder for unresolved llm-continuation path is documented and justified. |
| tests/recorded/rails/public_api/test_state.py | State round-trip across Colang 1.0 turns and per-call log isolation; fully deterministic with FakeLLMModel, clear documentation of known accumulate-vs-reset semantics difference. |
| tests/recorded/rails/public_api/configs.py | Adds RETRIEVAL_RAILS_CONFIG, STATE_CONFIG, OPENAI_TOOLS_CONFIG, OPENAI_KB_CONFIG, OPENAI_TRACING_CONFIG, TWO_INPUT_RAILS_CONFIG, and related model constants cleanly. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
subgraph "Recorded Test Suite (frozen cassettes)"
TG["test_generate.py\nOpenAI + NIM contract\ngenerate / generate_async"]
TS["test_stream.py\nStreaming parity matrix"]
TSC["test_stream_cancellation.py\nTeardown / partial-on-abort"]
TT["test_tools.py\nTool-call surfaces"]
TTR["test_tracing.py\nTraced vs untraced parity"]
TKB["test_kb.py\nEmbeddings + RAG retrieval"]
TO["test_options.py\nGenerationResponse shape"]
end
subgraph "Deterministic (FakeLLMModel, no cassette)"
TP["test_parity.py\nParity matrix\ninput/output/parallel × pass/block/modify"]
TC["test_concurrency.py\nContextvar isolation\nasyncio.gather concurrent vs sequential"]
TE["test_events.py\ngenerate_events_async"]
TST["test_state.py\nState round-trip + log isolation"]
TV2["test_colang_v2.py\nColang 2.x runtime"]
end
subgraph "Server Tests"
TSERV["test_chat_completions.py\nHTTP contract"]
SCONF["server/conftest.py\nTestClient fixture"]
end
subgraph "Live Smoke (auto-skip without real keys)"
TLS["test_live_smoke.py\nOpenAI + NIM + KB"]
end
CONFIGS["configs.py\nRailsConfigSource definitions"]
CONFIGS --> TG
CONFIGS --> TS
CONFIGS --> TSC
CONFIGS --> TP
CONFIGS --> TC
CONFIGS --> TE
CONFIGS --> TST
CONFIGS --> TV2
CONFIGS --> TKB
CONFIGS --> TLS
SCONF --> TSERV
TP -- "GENERATE_CELLS / ParityCell" --> TC
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
subgraph "Recorded Test Suite (frozen cassettes)"
TG["test_generate.py\nOpenAI + NIM contract\ngenerate / generate_async"]
TS["test_stream.py\nStreaming parity matrix"]
TSC["test_stream_cancellation.py\nTeardown / partial-on-abort"]
TT["test_tools.py\nTool-call surfaces"]
TTR["test_tracing.py\nTraced vs untraced parity"]
TKB["test_kb.py\nEmbeddings + RAG retrieval"]
TO["test_options.py\nGenerationResponse shape"]
end
subgraph "Deterministic (FakeLLMModel, no cassette)"
TP["test_parity.py\nParity matrix\ninput/output/parallel × pass/block/modify"]
TC["test_concurrency.py\nContextvar isolation\nasyncio.gather concurrent vs sequential"]
TE["test_events.py\ngenerate_events_async"]
TST["test_state.py\nState round-trip + log isolation"]
TV2["test_colang_v2.py\nColang 2.x runtime"]
end
subgraph "Server Tests"
TSERV["test_chat_completions.py\nHTTP contract"]
SCONF["server/conftest.py\nTestClient fixture"]
end
subgraph "Live Smoke (auto-skip without real keys)"
TLS["test_live_smoke.py\nOpenAI + NIM + KB"]
end
CONFIGS["configs.py\nRailsConfigSource definitions"]
CONFIGS --> TG
CONFIGS --> TS
CONFIGS --> TSC
CONFIGS --> TP
CONFIGS --> TC
CONFIGS --> TE
CONFIGS --> TST
CONFIGS --> TV2
CONFIGS --> TKB
CONFIGS --> TLS
SCONF --> TSERV
TP -- "GENERATE_CELLS / ParityCell" --> TC
Reviews (4): Last reviewed commit: "apply review suggestions" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Pouyanpi
force-pushed
the
test/recorded-tests-public-api-surface
branch
from
July 2, 2026 13:26
c82a5fb to
9490eb1
Compare
Adds recorded coverage for additional public-API surfaces (options, parity, kb, concurrency, events, state, tracing, tools, stream cancellation, colang 2.x) plus the server chat-completions suite and an embeddings-index determinism guard, on top of the library-rails stack. The colang 2.x self-check continuation test is marked xfail: a single generate_async on an llm-continuation flow returns an intermediate TimerBotAction polling state rather than the resolved answer.
Adds the missing test functions that consume the five stack-6 cassettes recorded but never referenced (B6.1 reasoning_content, B1.14 log.stats, B1.9 llm_output, A1.3 GenerationResponse shape, A3.10 streaming reasoning not inlined). Each replays offline under --block-network; snapshots filled from the committed cassettes. Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Pouyanpi
force-pushed
the
test/recorded-tests-public-api-surface
branch
from
July 13, 2026 11:54
9490eb1 to
9b291f7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds the recorded characterization baseline for the existing
LLMRailspublic API and server-facing behavior. This baseline is intended to support behavior-preserving review of the decomposition stack in #2123–#2131.What Changed
tests/recorded.Review Guidance
Use the LLMRails decomposition review matrix as the scenario index. It maps each characterized behavior to an exact test node and identifies which rows should be checked during each decomposition PR.
These tests intentionally characterize current behavior, including compatibility quirks. A failing row during #2123–#2130 should be treated as a potential behavior regression unless the stack explicitly identifies it as an intentional semantic change in #2131.
Related Issue(s)
This is maintainer-directed test infrastructure for the linked LLMRails decomposition stack.
Verification
AI Assistance
Checklist