Fix/tool call accuracy - #2300
Merged
Merged
Conversation
- equal check between pred_sequence and ref_sequence will check that both should be same length - when equal order of occurence of their elements are same
…with warning suppression
dosubot
Bot
added
the
size:L
edwiniac
pushed a commit
to edwiniac/ragas
that referenced
this pull request
Jan 31, 2026
…brantlabsai#2079) ## Problem ToolCallAccuracy returned scores > 1.0 when predicted tool calls contained duplicates (e.g., retries, multiple calls to the same tool). The previous fix (vibrantlabsai#2300) addressed this by using strict sequence matching, but introduced a regression where ANY length mismatch between predicted and reference tool calls resulted in a score of 0.0 — even when all reference calls were correctly made. ## Root Cause 1. `is_sequence_aligned` performed exact equality on full sequences, so extra predicted calls (retries) always caused alignment failure → score 0. 2. Positional zip-based scoring couldn't find best matches across tool calls. ## Fix ### Sequence Alignment - **Strict mode**: Reference names must be a subsequence of predicted names (preserves order requirement while tolerating extra/retried calls). - **Flexible mode**: Uses multiset containment (all reference tool names must appear in predicted with sufficient count). ### Scoring - Replaced positional zip with **greedy best-match**: for each reference tool call, finds the predicted call with the highest argument score. - Each predicted call is consumed after matching (no double-counting). - Final score clamped to [0.0, 1.0] as a safety net. ### Coverage Penalty - Only applied when predicted has FEWER calls than reference (missing tools). - Extra predicted calls no longer penalize the score. ## Test Results | Scenario | Before (v0.2.15) | Before (main) | After | |----------|------------------|---------------|-------| | 3 pred, 1 ref (retries) | 3.0 ❌ | 0.0 ❌ | 1.0 ✅ | | Exact retry (same args) | >1.0 ❌ | 0.0 ❌ | 1.0 ✅ | | Perfect match | 1.0 ✅ | 1.0 ✅ | 1.0 ✅ | | Wrong tool name | 0.0 ✅ | 0.0 ✅ | 0.0 ✅ | | Fewer pred than ref | 0.0 ✅ | 0.0 ✅ | 0.0 ✅ | Fixes vibrantlabsai#2079
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue Link / Problem Description