Skip to content

CHANGELOG.md

Latest commit

 

History

History
76 lines (58 loc) · 3.34 KB

CHANGELOG.md

File metadata and controls

76 lines (58 loc) · 3.34 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[1.3.0] - 2026-05-03

Added

  • Scenario-Check-Pass-Rate: New metric for special scenario validation, verifying model behavior in edge cases
    • Reference threshold: should be 100%
    • Added to baseline metrics and README evaluation tables

Changed

  • Updated M2.5/M2.7 baseline metrics with May 2026 data (100 concurrency, 10 rounds)

[1.2.0] - 2026-03-19

Added

  • Batch verification script (run_batch_sequential.sh): New script for running pass@10 verification tests
    • Supports 10 iterations by default for statistical significance
    • Auto-generates metrics reports and baseline comparisons
    • Parameters: --module, --url, --model, --api-key, --max-workers, --mm-model, etc.
  • Added expected_tool_call_total_count to summary statistics in validator/tool_calls.py
  • Added batch testing documentation to README.md and README_CN.md
  • New script dependencies:
    • scripts/batch_verify.py: Batch verification executor
    • scripts/calculate_batch_metrics.py: Metrics aggregation and calculation
    • scripts/compare_with_baseline.py: Baseline comparison tool
    • scripts/calculate_toolcall_similarity.py: Tool call similarity calculator

Changed

  • ToolCalls-Match-Rate denominator: Now uses expected_tool_call label count from sample.jsonl as the denominator for more accurate match rate calculation
  • All Chinese comments and outputs converted to English for better internationalization
  • calculate_batch_metrics.py now reads expected_tool_call statistics directly from sample.jsonl

Fixed

  • Fixed path consistency issues in shell script calling Python scripts

[1.1.0] - 2026-03-18

Changed

  • ToolCalls-Match-Rate redefined: Changed from simple "proportion of triggered tool calls" to a match rate based on expected labels
    • New formula: tool_calls_match_rate = (tool_calls_finish_tool_calls + stop_finish_stop) / expected_tool_call_total_count
    • i.e., proportion of cases where actual result matches expected result

Added

  • Added expected_tool_call label field to test set sample.jsonl, indicating whether each case is expected to trigger a tool call
  • Added confusion matrix statistics:
    • tool_calls_finish_tool_calls: expected tool_call, actual tool_call (True Positive)
    • tool_calls_finish_stop: expected tool_call, actual stop (False Negative)
    • stop_finish_tool_calls: expected stop, actual tool_call (False Positive)
    • stop_finish_stop: expected stop, actual stop (True Negative)
  • Added expected_tool_call field to results, recording the expected label from the original request

Fixed

  • Backward compatible with historical data without expected_tool_call label (incremental mode)

[1.0.0] - 2026-01-18

Changed

  • Stable release

[0.1.0] - 2025-11-18

Added

  • Initial release of MiniMax Provider Verifier
  • Support for multiple validators (ToolCalls, Russian Characters, Repeat N-Gram)
  • Concurrent request processing
  • Batch provider testing
  • Incremental mode for rerunning failed requests
  • Dynamic validator selection based on check_type
  • Detailed test reports and statistical summaries
  • Custom validator support