fix: Preserve tool calls for LLMRails tool rails

[JashG]

Description

This PR enables tool call guardrails to work when checking a tool-call message directly (as opposed to when the library itself generates the tool call).

• Preserve assistant messages with tool_calls when LLMRails runs with dialog rails disabled, so they are converted into BotToolCalls events and evaluated by tool output rails.
• Include tool input/output rail events in GenerationLog, so activated tool rails are reported with the correct type, name, duration, and stop handling.

Desired use case:
Agentic workflows may not rely on generate_async to orchestrate LLM calls end-to-end. Instead, the caller might invoke specific rails at different points in the request life cycle. Example:

• Agent generates a response that contains tool_calls,
• Before the agent executes the tools, use the library to validate the tool call name/args
• After the agent executes the tools out-of-band, use the library to validate the tool call results

Current issue:
In LLMRails, assistant messages were treated as plain text output and removed from the message history. If that assistant message contained tool_calls, the tool calls never became BotToolCalls events, so tool output rails could be skipped.

Related Issue(s)

#2096

Verification

AI Assistance

• No AI tools were used.
• AI tools were used; a human reviewed and can explain every change (tool: GPT 5.5 via Cursor).

Checklist

• I've read the CONTRIBUTING guidelines.
• This PR links to a triaged issue assigned to me.
• My PR title follows the project commit convention.
• I've updated the documentation if applicable.
• I've added tests if applicable.
• I've noted any verification beyond CI and any checks I couldn't run.
• I did not update generated changelog files manually.
• I addressed all CodeRabbit, Greptile, and other review comments, or replied with why no change is needed.
• @mentions of the person or team responsible for reviewing proposed changes.

Summary by CodeRabbit

• New Features

  • Tool-related rails now appear in generation logs as distinct entries, improving traceability for tool input and output handling.

• Bug Fixes

  • Assistant messages containing tool calls are now preserved in conversation history when needed, preventing tool-call information from being dropped.
  • Tool-call extraction now consistently uses the most relevant tool-call event when multiple event types are present.
  • Tool parameter checks now behave correctly with dialog rails disabled, including both allowed and refused tool-call cases.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR fixes tool-call guardrails for agentic workflows where the caller — not generate_async — is responsible for producing assistant messages. The two core changes are: (1) preserving assistant messages that contain tool_calls in the message history when dialog rails are disabled so they are converted into BotToolCalls events, and (2) updating extract_tool_calls_from_events to prefer the post-rail StartToolCallBotAction event over the pre-rail BotToolCalls fallback.

• llmrails.py: Adds not messages[-1].get(\"tool_calls\") to the guard that strips the trailing assistant message, keeping tool-call messages in the pipeline so they reach _get_events_for_messages and trigger tool output rails.
• processing_log.py: Extends compute_generation_log to handle StartToolOutputRail/StartToolInputRail start events and their corresponding Finished events, using the same state-machine pattern as existing input/output rails.
• utils.py + tests: extract_tool_calls_from_events now returns the first StartToolCallBotAction result (post-rail authoritative value) and falls back to the last BotToolCalls; new integration tests cover both the blocked and approved paths with dialog disabled.

Confidence Score: 5/5

Safe to merge; all changes are additive guards and logging extensions that do not affect existing paths.

The llmrails.py guard is a single boolean condition added to an existing branch — it only changes behavior when the trailing assistant message explicitly carries tool_calls, a case that was previously silently dropped. The processing_log.py additions mirror the well-established input/output rail state machine exactly. The extract_tool_calls_from_events change introduces a prioritized fallback covered by both new unit tests and existing integration tests.

tests/test_tool_calls_event_extraction.py — the local validate_tool_parameters fixture was not updated with the JSON-string-safe helper applied to the same function in test_tool_output_rails.py, and two new tests are unnecessarily declared async.

Important Files Changed

Filename	Overview
nemoguardrails/rails/llm/llmrails.py	Preserves assistant messages with tool_calls in message history when dialog rails are disabled, so they reach _get_events_for_messages and become BotToolCalls events; the guard is a clean, minimal condition addition.
nemoguardrails/actions/llm/utils.py	Updated extract_tool_calls_from_events to prefer the post-rail StartToolCallBotAction event and fall back to the last BotToolCalls; the early-return-on-first-match for StartToolCallBotAction is safe given each flow produces at most one such event.
nemoguardrails/logging/processing_log.py	Adds StartToolOutputRail/StartToolInputRail start events and ToolOutputRailFinished/ToolInputRailFinished finish events to the state machine, following the exact same pattern as input/output rails; also adds the four tool flow names to ignored_flows.
tests/test_tool_output_rails.py	Adds _tool_arguments helper to handle both dict and JSON-string arguments, fixes the existing validate_tool_parameters action, and adds two new integration tests covering the blocked and approved tool-call paths with dialog disabled.
tests/test_tool_calls_event_extraction.py	Adds two unit tests for extract_tool_calls_from_events; both tests use async def with @pytest.mark.asyncio but contain no async operations, and the local validate_tool_parameters fixture still uses raw func.get("arguments", {}) without the JSON-string-safe helper.
tests/test_logging.py	Adds a focused synchronous unit test for the new tool-rail entries in compute_generation_log, covering both tool_output and tool_input types, names, and durations.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Caller
    participant LLMRails
    participant _get_events_for_messages
    participant Runtime
    participant ToolOutputRail

    Caller->>LLMRails: "generate_async(messages=[user, assistant(tool_calls)], dialog=False)"
    Note over LLMRails: New guard: tool_calls present? YES keep assistant message
    LLMRails->>_get_events_for_messages: messages (assistant msg preserved)
    _get_events_for_messages-->>LLMRails: [..., BotToolCalls event]
    LLMRails->>Runtime: generate_events([..., BotToolCalls])
    Runtime->>ToolOutputRail: StartToolOutputRail
    ToolOutputRail-->>Runtime: ToolOutputRailFinished / abort
    Runtime-->>LLMRails: new_events (incl. StartToolCallBotAction or block)
    LLMRails->>LLMRails: extract_tool_calls_from_events prefers StartToolCallBotAction
    LLMRails-->>Caller: GenerationResponse(tool_calls or blocked message)

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Caller
    participant LLMRails
    participant _get_events_for_messages
    participant Runtime
    participant ToolOutputRail

    Caller->>LLMRails: "generate_async(messages=[user, assistant(tool_calls)], dialog=False)"
    Note over LLMRails: New guard: tool_calls present? YES keep assistant message
    LLMRails->>_get_events_for_messages: messages (assistant msg preserved)
    _get_events_for_messages-->>LLMRails: [..., BotToolCalls event]
    LLMRails->>Runtime: generate_events([..., BotToolCalls])
    Runtime->>ToolOutputRail: StartToolOutputRail
    ToolOutputRail-->>Runtime: ToolOutputRailFinished / abort
    Runtime-->>LLMRails: new_events (incl. StartToolCallBotAction or block)
    LLMRails->>LLMRails: extract_tool_calls_from_events prefers StartToolCallBotAction
    LLMRails-->>Caller: GenerationResponse(tool_calls or blocked message)

Loading

_{Reviews (3): Last reviewed commit: "Fix tests; address PR comments" | Re-trigger Greptile}

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f6c32115-e2cf-4aac-8fdd-ef5af1291d92

📥 Commits

Reviewing files that changed from the base of the PR and between 3b0b38e and e828b3f.

📒 Files selected for processing (5)

• nemoguardrails/logging/processing_log.py
• nemoguardrails/rails/llm/llmrails.py
• tests/test_logging.py
• tests/test_tool_calls_event_extraction.py
• tests/test_tool_output_rails.py

---

📝 Walkthrough

Walkthrough

Extends compute_generation_log to track tool_input/tool_output rails as first-class log entries. Fixes generate_async to preserve assistant messages containing tool_calls in conversation history when dialog rails are disabled. Adds unit tests for logging, tool call extraction preference, and tool output rail behavior.

Changes

Tool Rails Logging and History Fix

Layer / File(s)	Summary
Tool rail tracking in compute_generation_log nemoguardrails/logging/processing_log.py, tests/test_logging.py	Expands ignored_flows, adds StartToolOutputRail/StartToolInputRail event dispatch creating ActivatedRail entries, broadens rail-finished and stop/decision logic to cover tool_output/tool_input types, and tests the resulting activated_rails output.
Preserve assistant tool_calls in generate_async nemoguardrails/rails/llm/llmrails.py, tests/test_tool_output_rails.py	Gates the "move last assistant message to bot_message" path to skip messages with tool_calls, keeping them in history for downstream tool output rails; two new async tests verify refusal and passthrough behavior with dialog: False.
extract_tool_calls_from_events preference tests tests/test_tool_calls_event_extraction.py	Two new tests assert StartToolCallBotAction tool calls are returned correctly and preferred over an earlier BotToolCalls event.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~15 minutes

Possibly related issues

• bug: Preserve tool calls for LLMRails tool rails #2096: The fix to preserve assistant tool_calls messages in history and the new tool rail logging directly address the tool-rails bug described in this issue.

Suggested labels

size: L

Suggested reviewers

• cparisien
• Pouyanpi

🚥 Pre-merge checks | ✅ 4 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 18.18% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Results For Major Changes	⚠️ Warning	The PR is a substantial feature/fix, but the description leaves Verification empty and gives no actual test results or testing details.	Add a brief Verification section listing the tests run (or CI results) and any relevant outcomes/notes.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly reflects the main change: preserving tool calls in LLMRails for tool-rail processing.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[Pouyanpi]

Thanks @JashG, looks good. There is a gap right now that's why e828b3f tests are failing. when a tool-output rail passes, the validated tool calls are silently dropped from the response. can you double check and fix it ?

[JashG]

Thanks @JashG, looks good. There is a gap right now that's why e828b3f tests are failing. when a tool-output rail passes, the validated tool calls are silently dropped from the response. can you double check and fix it ?

Thanks for catching @Pouyanpi, this should be fixed.

[Pouyanpi]

Thanks @JashG, merging 👍🏻