[None][fix] Fix RPCStreamingError ('LlmResponse' object is not iterable) with num_postprocess_workers > 0

[shikicloud]

Dev Engineer Review

• Updated RpcWorker.fetch_responses to accept both individual LlmResponse objects and batched responses.
• Preserves existing batch handling with extend().
• Fixes the RPCStreamingError: 'LlmResponse' object is not iterable failure when postprocess workers are enabled.
• No public API or configuration changes.
• No test changes.

QA Engineer Review

No test changes.

Description

With num_postprocess_workers > 0, generation crashes at start with RPCStreamingError: 'LlmResponse' object is not iterable: the classic _send_rsp path enqueues a bare response (worker.result_queue.put(response)), but RpcWorker.fetch_responses assumes every queue item is a batch and calls extend() on it. Accept both shapes on the consumer side.

Test Coverage

Reproduced deterministically with num_postprocess_workers=4 (RPC/Ray orchestrator); no longer crashes with this patch. Existing executor unit tests cover the batched path.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[shikicloud]

/bot run

[coderabbitai]

Walkthrough

fetch_responses now accepts queue entries that contain either individual response objects or response batches.

Changes

Response queue handling

Layer / File(s)	Summary
Support individual and batched responses tensorrt_llm/executor/rpc_worker_mixin.py	fetch_responses appends individual responses and extends batched responses instead of assuming that every queue entry is iterable.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the RPCStreamingError fix and the condition involving num_postprocess_workers.
Description check	✅ Passed	The description explains the cause and solution, documents test coverage, and includes the required checklist.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

tensorrt_llm/executor/rpc_worker_mixin.py (1)

76-85: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add regression coverage for bare LlmResponse entries.

The existing tests cover batched responses, but this change adds support for a bare response. Enqueue one LlmResponse and verify that fetch_responses() returns a one-element list. Retain a batch case to cover both queue shapes.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tensorrt_llm/executor/rpc_worker_mixin.py` around lines 76 - 85, Add
regression coverage for the response-queue handling near fetch_responses: retain
the existing batched-response test and add a case that enqueues one bare
LlmResponse, then verifies fetch_responses() returns a one-element list
containing it.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tensorrt_llm/executor/rpc_worker_mixin.py`:
- Around line 76-85: Add regression coverage for the response-queue handling
near fetch_responses: retain the existing batched-response test and add a case
that enqueues one bare LlmResponse, then verifies fetch_responses() returns a
one-element list containing it.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 438d5035-f740-47ef-b02c-7975168bbcb1

📥 Commits

Reviewing files that changed from the base of the PR and between 67dd1b7 and 82e94f8.

📒 Files selected for processing (1)

• tensorrt_llm/executor/rpc_worker_mixin.py

[tensorrt-cicd]

PR_Github #65010 [ run ] triggered by Bot. Commit: 82e94f8 Link to invocation

[tensorrt-cicd]

PR_Github #65010 [ run ] completed with state SUCCESS. Commit: 82e94f8
/LLM/main/L0_MergeRequest_PR pipeline #52820 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[shikicloud]

/bot run --stage-list "DGX_B200-PyTorch-5"

[tensorrt-cicd]

PR_Github #65026 [ run ] triggered by Bot. Commit: 82e94f8 Link to invocation

[tensorrt-cicd]

PR_Github #65026 [ run ] completed with state SUCCESS. Commit: 82e94f8
/LLM/main/L0_MergeRequest_PR pipeline #52835 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation