feat(benchmark): add Annoy migration benchmarks

[Pouyanpi]

Summary

Adds benchmark tooling and reproduction notes used to compare the previous Annoy backend with the new exact NumPy backend.

What Changed

• Adds a benchmark harness for build time, search latency, recall, and memory.
• Adds benchmark methodology, reproduction notes, and captured baseline results.

Stack

Part 2 of 2.

• Base: drop-annoy-feature
• Head: drop-annoy-benchmark
• Previous: feat(embeddings): replace Annoy with exact NumPy search #1957

Review this PR against drop-annoy-feature; the implementation change is in the parent PR.

Validation

poetry run python benchmark/embedding_backend/bench_embedding_backend.py --sizes 100 --queries 5
passed, NumPy-only smoke run

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR adds a standalone benchmark harness (bench_embedding_backend.py) and accompanying README to document the build-time, latency, recall, and memory comparison between the previous Annoy backend and the new exact NumPy backend for BasicEmbeddingsIndex.

• Benchmark script generates deterministic clustered vectors, builds both indexes, measures p50/p95 search latency, recall@1 and recall@k against exact cosine ground truth, and prints a markdown table. Annoy is detected at runtime so the script degrades gracefully to NumPy-only when Annoy is not installed.
• README documents methodology, reproduction steps (including the native-build requirement that motivated dropping Annoy), and captured baseline results showing NumPy is faster at ≤10k vectors and exact (1.0 recall) at all sizes versus Annoy's 0.06 recall@1 at 100k.

Confidence Score: 5/5

Safe to merge — the change is entirely additive benchmark tooling with no modifications to production code.

Both new files are entirely additive with no touch to any production path. The script is self-contained, degrades gracefully without Annoy, and the logic is straightforward. The two findings are minor: a dead mean_ms field that is stored but never rendered, and a silent ground-truth truncation when n < k that cannot occur with the documented default parameters.

No files require special attention; both are new benchmark-only additions.

Important Files Changed

Filename	Overview
benchmark/embedding_backend/bench_embedding_backend.py	New benchmark harness for Annoy vs NumPy; two minor issues: mean_ms stored in result rows but never rendered in the markdown table, and exact_topk silently returns fewer than k columns when n < k causing misleading recall labels.
benchmark/embedding_backend/README.md	New documentation for the benchmark; methodology, reproduction steps, and captured results are clearly explained. The mem (MB) = 0.0 results are properly footnoted as platform-specific (non-Linux).

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[main] --> B[parse args]
    B --> C{annoy_available?}
    C -- yes --> D[BACKENDS = annoy + numpy]
    C -- no --> E[BACKENDS = numpy only]
    D --> F[run]
    E --> F
    F --> G[for each N in sizes]
    G --> H[make_normalized_vectors matrix]
    H --> I[make_normalized_vectors queries]
    I --> J[exact_topk ground truth]
    J --> K[for each backend]
    K --> L[builder → build_s, search_fn, mem_mb]
    L --> M[warm-up call]
    M --> N[time_searches → results, p50, p95, mean]
    N --> O[recall_at_k × 2]
    O --> P[append row dict]
    P --> K
    K -- next backend --> K
    G -- next N --> G
    F --> Q[print_markdown]

Loading

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
benchmark/embedding_backend/bench_embedding_backend.py:197
`mean_ms` — dead dict field

`mean` is captured from `time_searches` and stored in every result row as `mean_ms`, but `print_markdown` never reads it and the progress `print()` on line 203–207 doesn't print it either. This is the same pattern as the recently removed `rec5` field. Either render it in the markdown table (as an additional `search mean (ms)` column) or drop the field and the fourth return value from `time_searches` to keep the output and the stored data consistent.

### Issue 2 of 2
benchmark/embedding_backend/bench_embedding_backend.py:80-87
`exact_topk` silently truncates ground truth when `n < k`

The docstring promises shape `(Q, k)`, but when `n < k` the `argpartition` result has only `n` columns, and NumPy's permissive slicing means `[:, :k]` silently returns `(Q, n)` instead of raising. Downstream, `recall_at_k(results, truth, k)` then treats `n` items as the full ground-truth set, making both backends score 1.0 trivially. The markdown table would label the column `recall@20` while actually reporting something different. This doesn't trigger with the default sizes (≥100) and k=20, but a user passing e.g. `--sizes 5 10 --k 20` would get silently wrong recall numbers. A guard or assertion in `run()` would prevent the confusion.

_{Reviews (5): Last reviewed commit: "apply review suggestions" | Re-trigger Greptile}

[tgasser-nv]

This is great, thanks for doing the benchmarking to quantify the tradeoffs between annoy and numpy. I ran a regression on the Results table and the breakeven point is N ~= 5,000. I can't imagine anyone having more than 100 dialog rules, so this is well underneath the breakeven point.