You are an agent evolving a strategy through automated self-play.
evolve.py— core protocols and evolution loopratings.py— Bradley-Terry Elo ratingstracker.py— CLI for tracking and analysis
Each iteration:
- Check standings:
uv run tracker.py leaderboard - Pick a parent: choose a version from the Pareto front to branch from
- Analyze: look at win/loss patterns, replay traces, failure modes
- Mutate: create a new version with targeted improvements
- Evaluate: benchmark the new version against top opponents
- Record:
uv run tracker.py record vNew vOld --wins W --losses L - Reflect: did it improve? what to try next?
- Repeat
- One version per iteration. Name sequentially: v1, v2, v3, ...
- Minimum 100 games per matchup for statistical significance
- Match data is append-only — never delete results
- Always use
uv run tracker.py suggest vNto pick opponents — it uses information-theoretic scoring to find the most informative matchup. Do not manually cherry-pick opponents; this leads to sparse coverage and unreliable ratings. - Benchmark each new version against at least 3 opponents before drawing conclusions. Versions with fewer opponents are flagged with
?on the leaderboard. - Branch from Pareto-front versions, not just the highest-rated
- When stuck, analyze replays or traces to diagnose specific failure modes
- Commit matches.json after each significant benchmark round
To use this framework, fill in the sections below.
What arena do the strategies compete in?
(e.g., game simulator, prompt evaluation harness, model training benchmark)
What does a "version" look like?
(e.g., a Python file, a C++ binary, a prompt template, a config file)
How do you run a head-to-head benchmark?
(e.g., uv run benchmark.py --player-a vN --player-b vM -n 100)