feat(test): eval/benchmark suite for MCP tool retrieval quality

[singsing1]

feat(test): eval/benchmark suite for MCP tool retrieval quality

Summary

Adds a deterministic benchmark for the on-demand MCP tool-retrieval feature (agent/tools/mcp/tool_retrieval.py) that measures retrieval quality — not just functional correctness, which test_mcp_tool_retrieval.py already covers.

Relates to #2937 — the on-demand MCP tool-retrieval feature this benchmark evaluates. The feature shipped in #2940 with unit tests for correctness, but no way to measure how good the retrieval actually is. This PR fills that gap.

Today the project has no eval/benchmark anywhere (only a Playwright JS evaluate). This PR introduces a reproducible quality baseline for the retrieval path so that regressions in the embedding model / vector index can be tracked over time.

Motivation

select_mcp_tools decides which MCP tools get injected into each LLM turn. The existing unit tests verify the invariants (only-grows, fallback to full injection, dimension-mismatch handling) — i.e. "does it behave correctly?". But nothing answers: "is the retrieval actually any good?" — does it pick the right tools, and how many tokens does on-demand injection save versus injecting all of them?

Without this, a silent regression in embedding quality would go unnoticed until users complain. This PR closes that gap with a self-contained, byte-for-byte reproducible benchmark.

What it measures

On a fixed seed=42 fixture dataset (30 MCP tools in 5 semantic clusters, 12 queries):

Metric	Meaning
Precision@k	Of the top-k tools injected, how many were relevant?
Recall@k	Of the relevant tools, how many made it into top-k?
MRR	Where does the first relevant tool land in the ranking?
Token savings	full-injection tokens − on-demand top-k tokens (avg per query)

A sweep over top_k ∈ {3, 5, 8, 10, 15} surfaces the recall-vs-token-savings trade-off so maintainers can pick a sensible default.

Example output

top_k |    P@k |    R@k |    MRR | full tok | ondemand |  saved
---------------------------------------------------------------
    3 |  1.000 |  0.500 |  1.000 |     1182 |      118 |  90.0%
    5 |  1.000 |  0.833 |  1.000 |     1182 |      197 |  83.4%   <- sweet spot
    8 |  0.750 |  1.000 |  1.000 |     1182 |      313 |  73.5%
   10 |  0.600 |  1.000 |  1.000 |     1182 |      391 |  66.9%
   15 |  0.400 |  1.000 |  1.000 |     1182 |      588 |  50.2%

On this synthetic dataset top_k=5 is the sweet spot: 100% precision, 83% recall, ~83% tool-schema token savings versus injecting all MCP tools.

Dataset: why synthetic, and its limits

The fixture is synthetic, not captured from a real deployment — deliberately. Precision/recall require a ground truth (which tools are relevant for a given query), and CowAgent ships no such labelled set: real conversations don't come with "the correct tool subset" annotations. Constructing one is the only way to get IR metrics at all.

The construction is grounded, not arbitrary:

• The 5 semantic clusters map to real, common MCP tool categories (filesystem, web, database, shell, knowledge) — the actual tool families users connect via MCP servers.
• Within-cluster vectors are near (high cosine), across-cluster far — mirroring how a real embedding model clusters semantically-related tools.
• Token costs are estimated from the real {name, description, inputSchema} JSON shape CowAgent injects.

What this benchmark can and cannot claim:

• ✅ It validates the retrieval algorithm (cosine ranking, top-k selection, the only-grows union) on controlled, cluster-structured data, and gives a reproducible regression baseline. If the algorithm or its defaults regress, this catches it.
• ❌ It does not measure end-to-end quality with a specific real embedding model + real MCP tool catalogue — those numbers would require a labelled corpus that doesn't exist yet. Treat the absolute precision/recall figures as "algorithm sanity on synthetic data", not as field-measured accuracy.

How to run

python -m tests.eval_mcp_tool_retrieval         # human-readable report
python -m tests.eval_mcp_tool_retrieval --json  # machine-readable (CI)
python -m pytest tests/eval_mcp_tool_retrieval.py  # smoke checks

The pytest run only asserts metrics land in a sane band (recall ≥ 0.80, savings > 0, determinism) so a broken embedding pipeline fails CI without being overly brittle.

Backward compatibility

Pure additive — a single new file under tests/, zero changes to production code, zero behaviour change, no new runtime dependencies (uses random/math from the stdlib; numpy is optional and only used by the module under test, not by this benchmark). The fixture vectors are generated from a fixed seed, so every reported number is reproducible across machines and Python versions.

Why this matters

• Establishes the project's first retrieval-quality baseline — a measurable target for future embedding/index improvements.
• Catches embedding-pipeline regressions in CI before they reach users.
• Quantifies the value of on-demand retrieval (token savings) in a way that's easy to cite in discussions about the default top_k.

[zhayujie]

Thanks for the thoughtful PR and the detailed write-up — the honesty about the synthetic-data limitations is appreciated.

However, we won't be merging this one. The main issue is that the benchmark runs entirely on synthetically generated vectors (fixed clusters, no real embedding provider), so the metrics mostly reflect the data generator rather than actual retrieval quality — and it can't catch real embedding/index regressions since that pipeline is never exercised. The core cosine + top_k + union invariants are also already covered by test_mcp_tool_retrieval.py.

If we revisit retrieval eval later, we'd want it running against the real embedding provider on a realistic tool catalogue. Thanks again for the contribution!