feat: fair search benchmark (P1) and bounded self-improvement (P2) (#28)

P1: chronological dev/holdout episode splits shared across arms, a
uniform transaction-cost model, and five comparison arms (buy-and-hold,
random search, grid search, frozen agent, frozen agent + memory) with
leakage sentinels, future-dated-memory filtering, and missing outcomes
kept explicit rather than coerced to zero. Reports net return,
drawdown, turnover, search efficiency, and cross-seed variance, with
failures counted in success-rate denominators.

P2: a bounded outer loop that mutates one policy family (prompt/
generator allocation) with full mutation provenance (parent id, patch,
diagnosis, expected benefit, budget, rollback ref), dev/validation
selection, a single frozen final-holdout grade guarded against reuse,
a fixed promotion epsilon plus protected-episode regression check, and
a random-mutation baseline under an identical budget. Ties/losses keep
the incumbent; persisting a config alone does not count as promotion.

Author: OnePunchMonk

README.md

@@ -297,6 +297,55 @@ python scripts/benchmark_harness_evolution.py \
 # Output: JSON report based on a mock fitness function (not backtests)
 ```
 
+### Fair Search Benchmark (P1) and Bounded Self-Improvement (P2)
+
+```bash
+# P1: compare arms (fixed / random_search / grid_search / frozen_agent /
+# frozen_agent_memory) on identical chronological dev/holdout episode
+# splits, with a uniform transaction-cost model and >=3 seeds per arm.
+python scripts/fair_search_benchmark.py --episodes 3 --seeds 7 11 19 \
+  --output results/fair_search_benchmark.json
+
+# P2: outer-loop policy mutation (mutates prompt_template/prompt_context)
+# evaluated on dev episodes, selected on a validation episode, promoted
+# only if it clears a fixed Sharpe-improvement threshold without regressing
+# on a protected episode, then frozen-graded once on final holdout episodes.
+# Also runs a random-mutation baseline and memory-disabled/shuffled-memory
+# ablations under the identical budget.
+python scripts/bounded_self_improvement.py --episodes 6 --seeds 7 11 19 \
+  --n-mutations 3 --output results/bounded_self_improvement.json
+```
+
+Both scripts run offline on deterministic synthetic OHLCV data by default
+(no API keys needed) and are development benchmarks, not claims about live
+or historical trading performance.
+
+- **Arms** (`src/agent/search_arms.py`): `fixed` is buy-and-hold; `random_search`
+  and `grid_search` are non-agent baselines over the momentum parameter grid;
+  `frozen_agent` runs the existing propose→backtest→reflect loop
+  (`src/agent/agent_graph.py`) once per episode with a fresh, empty memory
+  snapshot each time; `frozen_agent_memory` gives that same loop read access
+  to memory written by strictly earlier episodes only (never future ones —
+  see `filter_visible_memory`).
+- **Episode splits** (`src/agent/episode_splits.py`): chronological
+  (dev-window, sealed-holdout-window) pairs generated once and persisted to
+  JSON so every arm is graded on identical windows. A fixed bps-per-trade
+  transaction cost (`apply_transaction_costs`) is applied uniformly.
+- **Reported per arm/episode**: held-out net return after costs, max
+  drawdown, turnover, search efficiency (return per attempted candidate),
+  cross-seed mean/std, and full candidate logs (including failed attempts,
+  which count in the denominator of the success rate). Missing/failed
+  outcomes are reported as `"missing"`, never coerced to 0.
+- **Promotion criterion** (`src/agent/policy_mutation.py`): a candidate
+  policy is only promoted over the incumbent if its mean validation-episode
+  holdout Sharpe beats the incumbent's by more than `PROMOTION_EPSILON`
+  (0.10) AND it doesn't regress by more than `MAX_PROTECTED_REGRESSION`
+  (0.25) on a reserved protected episode. Ties, losses, and inconclusive
+  deltas keep the incumbent — persisting a new config is never itself
+  treated as improvement. The final holdout episodes can only be used for
+  one frozen grading pass per run (`FinalHoldoutGuard` errors loudly on
+  reuse).
+
 ### Run Agent (Streamlit UI)
 
 ```bash

scripts/bounded_self_improvement.py

@@ -0,0 +1,141 @@
+#!/usr/bin/env python3
+"""P2 -- Bounded self-improvement runner.
+
+Ties the inner loop (src.agent.agent_graph.run_agent, invoked per-episode
+under a given policy) to the outer loop (src.agent.policy_mutation): propose
+N candidate policy mutations, evaluate them on development episodes, select
+the best on a separate validation episode slice, gate promotion on a fixed
+threshold + protected-episode regression check, then do ONE final frozen
+evaluation of the selected policy on held-out final episodes.
+
+Also runs a random-mutation baseline under the identical budget, plus
+memory-disabled / shuffled-memory ablations, for comparison.
+
+Usage:
+    python scripts/bounded_self_improvement.py --n-mutations 3 --seeds 7 11 19
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import random
+import sys
+import tempfile
+from pathlib import Path
+from typing import Optional
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from src.agent.episode_splits import get_or_build_episodes, synthetic_ohlcv  # noqa: E402
+from src.agent.harness_config import harness_v1_base  # noqa: E402
+from src.agent.policy_mutation import FinalHoldoutGuard, run_bounded_self_improvement  # noqa: E402
+from src.agent.search_arms import _run_agent_offline, slice_dev  # noqa: E402
+
+ASSET = "SIM"
+
+
+def make_eval_fn(ohlcv, memory_mode: str, cost_bps: float, max_iterations: int):
+    """memory_mode: 'normal' | 'disabled' | 'shuffled'.
+
+    'disabled' gives every episode a fresh empty memory db (ablation: no
+    cross-episode evidence at all). 'shuffled' reuses one memory db but
+    episodes are evaluated in a shuffled (non-chronological) order so any
+    stored rows are misattributed to the wrong episode context -- isolating
+    whether real (correctly-timed) evidence, vs. just "more context", drives
+    any measured gain.
+    """
+    shared_dirs: dict = {}
+
+    def eval_fn(policy, episode, seed) -> Optional[float]:
+        dev_ohlcv = slice_dev(ohlcv, episode)
+        if memory_mode == "disabled":
+            with tempfile.TemporaryDirectory() as tmp:
+                state = _run_agent_offline(dev_ohlcv, episode.asset, seed,
+                                            str(Path(tmp) / "memory.db"), max_iterations=max_iterations)
+        else:
+            key = seed if memory_mode == "normal" else f"shuffled-{seed}"
+            if key not in shared_dirs:
+                shared_dirs[key] = tempfile.mkdtemp()
+            state = _run_agent_offline(dev_ohlcv, episode.asset, seed,
+                                        str(Path(shared_dirs[key]) / "memory.db"), max_iterations=max_iterations)
+        best = state.get("best_result") or {}
+        return best.get("sharpe")
+
+    return eval_fn
+
+
+def run_condition(label: str, ohlcv, episodes, args, use_random_baseline: bool, memory_mode: str) -> dict:
+    n = len(episodes)
+    if n < 4:
+        raise SystemExit("Need at least 4 episodes to split dev/val/protected/final.")
+    dev_episodes = episodes[: n // 2]
+    val_episodes = [episodes[n // 2]]
+    protected_episode = episodes[max(0, n // 2 - 1)]
+    final_episodes = episodes[n // 2 + 1:]
+    if not final_episodes:
+        final_episodes = [episodes[-1]]
+
+    eval_fn = make_eval_fn(ohlcv, memory_mode, args.cost_bps, args.max_iterations)
+    guard = FinalHoldoutGuard(path=Path(args.results_dir) / f"final_holdout_used_{label}.json")
+
+    result = run_bounded_self_improvement(
+        incumbent=harness_v1_base(),
+        dev_episodes=dev_episodes,
+        val_episodes=val_episodes,
+        final_episodes=final_episodes,
+        protected_episode=protected_episode,
+        eval_fn=eval_fn,
+        seeds=args.seeds,
+        n_mutations=args.n_mutations,
+        holdout_guard=guard,
+        use_random_baseline=use_random_baseline,
+        rng_seed=hash(label) % (2 ** 31),
+    )
+    result["label"] = label
+    print(f"[{label}] promote={result['promotion_decision']['promote']} "
+          f"reason={result['promotion_decision']['reason']}")
+    return result
+
+
+def main() -> None:
+    p = argparse.ArgumentParser()
+    p.add_argument("--episodes", type=int, default=6)
+    p.add_argument("--seeds", type=int, nargs="+", default=[7, 11, 19])
+    p.add_argument("--n-mutations", type=int, default=3)
+    p.add_argument("--cost-bps", type=float, default=5.0)
+    p.add_argument("--max-iterations", type=int, default=2)
+    p.add_argument("--splits-path", default="experiments/bounded_improvement_splits.json")
+    p.add_argument("--results-dir", default="experiments")
+    p.add_argument("--output", default="results/bounded_self_improvement.json")
+    args = p.parse_args()
+
+    ohlcv = synthetic_ohlcv(seed=args.seeds[0], n_days=3200, asset=ASSET)
+    episodes = get_or_build_episodes(
+        ohlcv, ASSET, path=Path(args.splits_path),
+        n_episodes=args.episodes, dev_days=320, holdout_days=50,
+    )
+
+    report = {
+        "diagnosed_mutation": run_condition("diagnosed", ohlcv, episodes, args,
+                                             use_random_baseline=False, memory_mode="normal"),
+        "random_mutation_baseline": run_condition("random_baseline", ohlcv, episodes, args,
+                                                    use_random_baseline=True, memory_mode="normal"),
+        "memory_disabled_ablation": run_condition("memory_disabled", ohlcv, episodes, args,
+                                                    use_random_baseline=False, memory_mode="disabled"),
+        "shuffled_memory_ablation": run_condition("shuffled_memory", ohlcv, episodes, args,
+                                                    use_random_baseline=False, memory_mode="shuffled"),
+    }
+    report["promotion_epsilon_and_protected_check"] = (
+        "See src/agent/policy_mutation.py: PROMOTION_EPSILON, MAX_PROTECTED_REGRESSION. "
+        "Inconclusive/tied/losing candidates keep the incumbent."
+    )
+
+    out_path = ROOT / args.output
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    out_path.write_text(json.dumps(report, indent=2, default=str))
+    print(f"Saved {out_path}")
+
+
+if __name__ == "__main__":
+    main()

scripts/fair_search_benchmark.py

@@ -0,0 +1,163 @@
+#!/usr/bin/env python3
+"""P1 -- Fair search benchmark.
+
+Runs several strategy-selection "arms" (fixed / random_search / grid_search /
+frozen_agent / frozen_agent_memory) over identical, frozen chronological
+episode splits (development window for search, sealed holdout window for
+final grading only), with a uniform transaction-cost model, at least 3 seeds
+per arm, and a full candidate log per episode for selection-bias analysis.
+
+Usage:
+    python scripts/fair_search_benchmark.py --episodes 3 --seeds 7 11 19 \\
+        --output results/fair_search_benchmark.json
+
+This is a development benchmark on deterministic synthetic fixtures, not a
+claim about live or historical trading performance. Runs offline (no LLM/API
+keys) by default via the agent's existing offline fallback path.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import tempfile
+from pathlib import Path
+from typing import Any, Dict, List
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from src.agent.episode_splits import (  # noqa: E402
+    DEFAULT_COST_BPS, get_or_build_episodes, synthetic_ohlcv,
+)
+from src.agent.search_arms import (  # noqa: E402
+    run_fixed_arm, run_frozen_agent_arm, run_frozen_agent_memory_arm,
+    run_grid_search_arm, run_random_search_arm,
+)
+
+ASSET = "SIM"
+
+
+def _summarize(results: List[dict]) -> Dict[str, Any]:
+    ok = [r for r in results if r["status"] == "ok"]
+    missing = [r for r in results if r["status"] != "ok"]
+    n_total = len(results)
+    n_candidates = sum(r["n_candidates"] for r in results)
+    n_failed_candidates = sum(r["n_failed"] for r in results)
+    tool_calls = sum(r["tool_calls"] for r in results)
+    wall_time = sum(r["wall_time_s"] for r in results)
+
+    net_returns = [r["holdout_net_return"] for r in ok]
+    drawdowns = [r["holdout_max_drawdown"] for r in ok]
+    turnovers = [r["holdout_turnover"] for r in ok]
+
+    def mean_std(xs):
+        if not xs:
+            return None, None
+        m = sum(xs) / len(xs)
+        var = sum((x - m) ** 2 for x in xs) / len(xs)
+        return m, var ** 0.5
+
+    net_mean, net_std = mean_std(net_returns)
+    dd_mean, dd_std = mean_std(drawdowns)
+    to_mean, to_std = mean_std(turnovers)
+
+    search_efficiency = None
+    if net_mean is not None and n_candidates > 0:
+        search_efficiency = net_mean / n_candidates
+
+    return {
+        "n_episode_seed_runs": n_total,
+        # success rate counts failed attempts in the denominator, per issue reqs
+        "success_rate": len(ok) / n_total if n_total else None,
+        "n_missing": len(missing),
+        "net_return_mean": net_mean,
+        "net_return_std": net_std,
+        "max_drawdown_mean": dd_mean,
+        "max_drawdown_std": dd_std,
+        "turnover_mean": to_mean,
+        "turnover_std": to_std,
+        "search_efficiency_return_per_candidate": search_efficiency,
+        "total_candidates_attempted": n_candidates,
+        "total_candidates_failed": n_failed_candidates,
+        "total_tool_calls": tool_calls,
+        "total_wall_time_s": wall_time,
+    }
+
+
+def main() -> None:
+    p = argparse.ArgumentParser()
+    p.add_argument("--episodes", type=int, default=3)
+    p.add_argument("--seeds", type=int, nargs="+", default=[7, 11, 19])
+    p.add_argument("--cost-bps", type=float, default=DEFAULT_COST_BPS)
+    p.add_argument("--max-iterations", type=int, default=2, help="agent arms only")
+    p.add_argument("--splits-path", default="experiments/fair_benchmark_splits.json")
+    p.add_argument("--output", default="results/fair_search_benchmark.json")
+    p.add_argument("--skip-agent-arms", action="store_true",
+                    help="skip frozen_agent/frozen_agent_memory (faster, no agent_graph dependency)")
+    args = p.parse_args()
+
+    if len(args.seeds) < 3:
+        raise SystemExit("Need at least 3 seeds for cross-seed variance reporting.")
+
+    data_seed = args.seeds[0]
+    ohlcv = synthetic_ohlcv(seed=data_seed, n_days=1500, asset=ASSET)
+    episodes = get_or_build_episodes(
+        ohlcv, ASSET, path=Path(args.splits_path),
+        n_episodes=args.episodes, dev_days=220, holdout_days=60,
+    )
+    if not episodes:
+        raise SystemExit("Not enough synthetic data to build even one episode; increase n_days.")
+
+    report: Dict[str, Any] = {"episodes": [e.to_dict() for e in episodes], "arms": {}}
+
+    def run_arm(name: str, fn) -> None:
+        rows = []
+        for ep in episodes:
+            for seed in args.seeds:
+                res = fn(ohlcv, ep, seed, args.cost_bps)
+                rows.append(res.to_dict())
+        report["arms"][name] = {"episode_results": rows, "summary": _summarize(rows)}
+        print(f"[{name}] {report['arms'][name]['summary']}")
+
+    run_arm("fixed", run_fixed_arm)
+    run_arm("random_search", run_random_search_arm)
+    run_arm("grid_search", run_grid_search_arm)
+
+    if not args.skip_agent_arms:
+        run_arm("frozen_agent", lambda oh, ep, seed, cb: run_frozen_agent_arm(
+            oh, ep, seed, cb, max_iterations=args.max_iterations))
+
+        # frozen_agent_memory: one persistent memory db shared across
+        # episodes IN CHRONOLOGICAL ORDER, reset once per seed so results
+        # are comparable across seeds and never see another seed's memory.
+        for seed in args.seeds:
+            with tempfile.TemporaryDirectory() as tmp:
+                memory_db = str(Path(tmp) / "memory.db")
+                for ep in episodes:  # episodes list is already chronological
+                    res = run_frozen_agent_memory_arm(
+                        ohlcv, ep, seed, args.cost_bps, memory_db_path=memory_db,
+                        max_iterations=args.max_iterations,
+                    )
+                    report["arms"].setdefault("frozen_agent_memory", {"episode_results": []})
+                    report["arms"]["frozen_agent_memory"]["episode_results"].append(res.to_dict())
+        rows = report["arms"]["frozen_agent_memory"]["episode_results"]
+        report["arms"]["frozen_agent_memory"]["summary"] = _summarize(rows)
+        print(f"[frozen_agent_memory] {report['arms']['frozen_agent_memory']['summary']}")
+
+    report["cost_model"] = {"cost_bps": args.cost_bps}
+    report["seeds"] = args.seeds
+    report["note"] = (
+        "Negative or tied results for adaptive arms are reported as-is, not suppressed. "
+        "Missing/failed outcomes are reported as status='missing' and are not coerced into "
+        "a fallback numeric score."
+    )
+
+    out_path = ROOT / args.output
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    out_path.write_text(json.dumps(report, indent=2, default=str))
+    print(f"Saved {out_path}")
+
+
+if __name__ == "__main__":
+    main()