lots of fixes in resend, downsize, summrize relearn analyzers and leading with avoidable waste on ui - #597
Merged
Conversation
`downsize` contributed $4.80 of a $1,191 Review-inbox rollup on a real 30-day coding-agent corpus. The reflex diagnosis was that its structural gate (input < 5,000 AND output < 500 AND tool_calls <= 5, summed per session) is unreachable for a coding-agent session, so the fix would be to loosen it. Measured first, and that diagnosis is wrong: 231 of 2,258 sessions (10.2%) clear the gate. They carry $6.74 of $8,222.53 in main-thread spend, 0.082%. The gate fires fine — small sessions are simply cheap. Loosening it would have produced more near-worthless cards and changed the dollars by nothing. So the thresholds are unchanged and that case is kept, demoted to secondary. The gap was a question nothing measured: a premium-tier model acting as the DRIVER of a long session, doing the reads and edits inline instead of routing them to cheap workers. Because every inline turn re-reads the whole accumulated context, that waste GROWS with session size — so a summed per-session gate becomes less likely to fire the worse the problem gets. The new primary case flags a session where the main-thread driver is premium-tier, no subagent is ever dispatched, context passes 50K tokens and at least 10 main-thread tool calls ran. It prices the inline-vs-routed counterfactual over the turns inside a tool-driven stretch, in two exact halves: the re-read tail that stops happening once the material lives in a worker's own context, and the same turns' work repriced at the substitute worker tier (named in the basis string). The premium driver itself is left in place and never repriced — an interactive session has to stay smart, so nothing here asks anyone to downgrade their own thread. Disjointness (Critical Rule 27) is the main correctness risk, because `resend` already prices the same mechanism. The two are partitioned by session population behind one shared predicate, `premium_driver_role`, which both analyzers import from the new `analyzers/resend_tail.py` alongside the tail arithmetic itself — so neither side can reimplement it slightly differently, and the populations cannot drift. Against `subagent` the case is disjoint by definition: a flagged session dispatches no subagent, so it carries no subagent spans at all. On the same corpus: downsize $4.80 -> $568.77, resend $741.47 -> $405.86 (it hands over exactly the 748 driver sessions and says so on its payload), rollup $1,191.74 -> $1,420.09. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…urfaces The backend computes the strongest number this product produces and no surface renders it. `cost_of_waste_usd` is priced per token class at the rates that actually billed, carries a long honesty basis, ships on the cost-proposals payload, and `grep -c cost_of_waste ui/index.html` returned 0. A user reading the Review inbox saw $1,152 recoverable and had no idea re-sent context had already cost them $6,973. Replaces the forward-looking headline with a backward-looking one. A projected figure depends on how many sessions the user runs next month, which is unknowable; an observed figure is checkable against a bill they already paid, and costs no trust to accept. What lands: * `CostProposal.past_overspend_usd/_tokens/_basis`, stamped in ONE place (`_with_past_overspend`). A finding with its own observed cost of the flagged behaviour (today only resend, whose avoidable share is gated by a MEASURED offloadable share) leads with that and carries the avoidable share on `past_avoidable_*`. For the other six analyzers the window recoverable figure already IS the observed window overspend, verified per analyzer, so it is carried across unchanged and there is no second number: rendering both would show one quantity twice. * `past_overspend_rollup` + a `past_overspend` block on GET /relearn/cost-proposals, computed once over the open set. The Dashboard hero and the Review inbox headline read that one block, so they cannot disagree. Structurally separate from the recoverable rollup: neither function reads the other's fields, so no code path can sum them. * Never paced. The central 30-day ratio is a forward projection; the rollup takes no active-days/sessions argument, so there is nothing to project from. * Every label is past tense and "recoverable" appears on neither surface. The resend card states two past-tense sentences, not a ratio: a ratio reintroduces recovery language on a surface whose premise is that it claims nothing. * reuse's headline moves to avg cost x (reps - 1). The first planning call in a cluster had nothing to reuse yet, so charging for it states a figure a user can disprove; on a 2-repetition cluster it was a 2x overclaim. * The hero fetches on its own effect. Folded into the triage Promise.all it sat behind a loading shimmer for minutes waiting on a 30-day analyzer sweep while its own data was a 0.2s stored-block read. Tests assert the three properties that make the figure safe to show: never summed into a recoverable total, never paced, never re-derived in JS. Validated against a 359 MB copy of a real corpus through the real UI: hero and headline both $7,423.93 / 14.4B tokens from one payload, resend alone paired with "About $701.33 of that was avoidable", recoverable rollup unchanged at $1,152.46, "Do NOT read this as a saving" reachable from the card. Full suite before/after: identical failure sets, +16 passing. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
|
| Filename | Overview |
|---|---|
| tokenjam/api/routes/relearn.py | Replaces the former projected rollup response with the canonical past-overspend block and updates proposal-state filtering. |
| tokenjam/core/optimize/cost_proposals.py | Consolidates analyzer cards and aggregates around observed avoidable spend and separately reported behavioral cost. |
| tokenjam/core/optimize/analyzers/relearn.py | Expands relearn’s historical corpus and separates observed recurrence cost from fix availability. |
| tokenjam/core/transcript_sync.py | Adds continuous and startup transcript synchronization support. |
| tokenjam/ui/index.html | Updates dashboard and Review surfaces to lead with observed avoidable spend and separate cost-only figures. |
Reviews (15): Last reviewed commit: "feat(inbox): overspend as a tile beside ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Rework downsize around the model-ROLE, not just model price
… footnote summarize was excluded from COST_ANALYZERS entirely: it has its own curate/diff/apply surface, it is the denominator the write-budget spends against, and the standing rule was to consolidate cards rather than add them. Those are real distinctions, but the consequence was never argued — the second-largest figure this product computes (behind only resend) never reached the Review inbox headline a user actually reads. The Review inbox is the complete index of everything actionable, not the list of things whose apply flow happens to live here. Where a fix is APPLIED is a routing detail, not a reason to keep a measured figure off the one surface a user checks for "what's outstanding." So summarize now gets a normal card like every other analyzer (title reads "Review N oversized files, $X"), on the same observed window basis, reaching the same past-overspend headline every other card does — but its one affordance links into the tj summarize curate -> diff -> apply screen instead of an inline Apply/"Mark applied" action, since the underlying fix is a reviewed rewrite (structure kept, prose compressed) rather than a one-click change. The write-budget coupling that motivated the old exclusion is unaffected: it reads the summarize finding directly off the report, not off this proposal list, so the card never enters that netting pass itself (it is never apply_capable). The prior link-only "$X more, not summed here" disclosure mechanism is retired for summarize specifically; the generic plumbing stays in place for a future analyzer that might need it. Measured against a real 30-day corpus: summarize contributes one card ($4,126.59, 17 files) and correctly lands as the second-largest line in the past-overspend breakdown behind resend, ahead of downsize/deadweight/ subagent/reuse -- confirming the inbox-volume concern is a non-issue (one row, not one per file) and that the headline total now accounts for it. New tests pin: the card's shape and copy (never a bare "Apply"), that its window figure reaches the same past-overspend rollup as every other analyzer, and a regression guard asserting every COST_ANALYZERS member that produces a priced finding shows up in that rollup's breakdown. 🤖 shipped by [shiploop](https://github.com/anshss/shiploop) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`resend` computed its cost figure over every session with repeat volume ($6,972.80 on the reference corpus) and its avoidable figure over a filtered subset ($398.41), then rendered the first as the past-tense hero. Nobody wrote the claim down, but the pairing made it: that 94.3% of that money had been shown to be unavoidable. It had not. Measured, the gap decomposes as $1,787.15 (25.6%) in 706 sessions ceded whole to `downsize`'s driver-role case (analysed on another card, per Critical Rule 27), $13.54 in 446 sessions dropped by MIN_SESSION_CONTEXT_TOKENS before the calculation ran, and the remainder outside the compaction-bounded tail definition or below the measured offloadable share. Three ways of not being analysed; zero findings of necessity. Waste is only ever what could have been avoided, the same rule `reuse` already applies by pricing reps - 1 rather than reps. What changed: * `_with_past_overspend` now stamps `estimated_recoverable_usd` onto `past_overspend_*` for every analyzer, so the headline has one meaning across cards and the rollup is a coherent sum. The total observed cost, where an analyzer computes one, moves to new `observed_cost_*` fields carrying cost wording only, never summed with the headline (the avoidable figure is a subset of it) and reported as its own separate rollup key. * `context_resend` partitions its cost by the SAME predicate the avoidable figure uses (`_coverage_class`), so the coverage the card states cannot disagree with the coverage the code applied. The three buckets re-sum to the cost figure exactly. `offload_ceiling_usd` separates "outside the tail definition" from "discounted by the measured share". * A `coverage_note` states the split in prose and ends by saying the difference between the two figures is what was NOT ANALYSED, not what was unavoidable. * `offloadable_share` now discloses its behavioural basis, sample size and spread in the basis string: measured across the 175 of 2,291 sessions (7.6%) that delegate at all, applied to the 92.4% that never do, per-session spread 0.001-1.0. A structural measure would be better, but the per-tool-call delegability it needs is computed nowhere in this tree, so the basis says so rather than leaving a bare scalar that reads like a corpus property. * Both sides of the paired display read the same raw 30-day window. No pacing ratio touches either, so their difference is attributable to avoidability alone and never to a time-basis artifact. * UI copy leads with "What you could have avoided" and renders the total as an explicit cost line plus the coverage note. Verified end to end against a read-only copy of a real 359 MB telemetry store: the new instrumentation reproduces the audit's independently-measured figures exactly (cost $6,972.802379, avoidable $398.414906, driver-role $1,787.154876 over 706 sessions, 446 sessions below the floor, share 0.4021 from 175 of 2,291, median 0.3339), then drove the real dashboard and Review inbox in a headless browser to confirm both surfaces render the corrected framing. Full suite 3742 passed, ruff and mypy clean. Co-Authored-By: Claude <noreply@anthropic.com>
…d cost relearn is the one analyzer whose entire signal is long-horizon recurrence, and it read Claude Code's on-disk transcripts -- which Claude Code rotates (cleanupPeriodDays). Measured on a real corpus it saw 37.4 days against a database holding 90+, so it was structurally incapable of accumulating any history however long tokenjam ran. The product's whole proposition is that it retains what the agent discards; an analyzer reading the discarded source cannot deliver it. Four changes, all measured against the real local corpus: * ARCHIVE LANE. Coding sessions tokenjam still holds telemetry for but whose transcript has been rotated away are now recovered from the spans table (coarser signatures, the same trade the OTel lane already makes). The two lanes are disjoint by session id, so nothing is counted twice. Bounded by storage.retention_days rather than the report window. Measured: 1,037 sessions and 1,679 failures that were previously invisible; corpus span 37.4 -> 43.5 days. Sentinel 1970-01-01 timestamps are now rejected by the span derivation, which widening the horizon would otherwise have exposed. * AN OBSERVED COST SURVIVES EVERY GATE. Three separate gates turned real, incurred cost into $0: no fix template matched, the write budget judged the rule net-negative, and the recurrence threshold. None of them establishes that the waste was avoidable -- they establish that we have no remedy, which is a gap in the product. Clusters now carry past_overspend_* (backward, observed, ungated) alongside estimated_* (forward, gated, netted). Measured: 46 of 55 clusters claim nothing and now report the $8.44 they actually cost, where they previously reported $0 on every basis. * NO FUTURE COST NETTED OUT OF A PAST FIGURE. A rule's standing cost is legitimately subtracted from a forward "is this worth doing" figure and never from money already spent. The past figure no longer moves when the write budget's basis does. * THE ROLLUP. relearn produced RelearnClusters, never CostProposals, so past_overspend_rollup and the Dashboard hero were blind to the entire self-improve loop. One aggregate card now carries the observation into the rollup ($46.19 where it used to contribute nothing). It states no forward claim: relearn's re-read tail is the same re-sent context the resend analyzer already claims, and two analyzers claiming one span in the recoverable rollup is the disjointness rule. The claim stays where it always was, on relearn's own per-cluster rows, counted once. The recurrence gate's residue (1,410 occurrences, $5.07) is now counted and priced on the finding rather than silently dropped -- not claimable, since a one-off failure is not a relearn, but not free either. Tests cover an episode older than transcript retention still counting, a no-fix-template cluster still carrying a cost, a net-negative rule not zeroing that cost, the past figure's independence from any standing-cost basis, the sentinel guard, and the rollup contribution. Co-Authored-By: Claude <noreply@anthropic.com>
Combines relearn's DB-horizon + action-availability-gate fix (base) with the observed_cost/avoidable field split for the waste headline (this branch). Resolved CLAUDE.md (renumbered the two independently-appended lessons) and tests/unit/test_past_overspend.py (kept both sides' new tests/helper, which land in the same insertion point). Also fixes a real interaction bug surfaced by the merge: _with_past_overspend() unconditionally bailed out when a proposal had no avoidable claim at all, so relearn's cost-only card (no forward claim, per rule 27) never got its cost_of_waste_* stamped onto observed_cost_*. Split the stamping into independent avoidable/cost branches so a cost-only card still surfaces its observed figure. Updated tests/unit/test_relearn_archive_and_cost.py's two assertions that were written against the pre-merge field semantics to match. Co-Authored-By: Claude <noreply@anthropic.com>
Completeness of the local store depended on a person remembering to run `tj backfill claude-code`. There were exactly two ingest paths and neither ran on its own: the live OTLP path (Claude Code's exporter has no retry and no buffer, so a session whose shell lacked the telemetry env vars, or that ran while the daemon was down or pointed at a dead port, is dropped permanently) and the on-demand backfill CLI. Nothing scheduled the latter, nothing caught up on daemon start, and nothing ever noticed the difference. That inverts the product's premise. Claude Code prunes its own transcripts after roughly 30 days, so every session missed at ingest is unrecoverable once that happens, and every analyzer figure is quietly understated until then. Measured on a real 5,615-file corpus, deduping on each transcript's INTERNAL sessionId rather than its filename (2,890 of those files live in nested subagents/ folders and carry their PARENT's session id, which is why two earlier measurements of this same gap disagreed by an order of magnitude): 2,710 distinct sessions on disk, 2,330 ingested, 380 absent. 155 of the 380 parse to zero spans -- a session that ended before its first model call has no usage to record and is correctly skipped -- leaving 225 genuinely missing sessions carrying 22,882 spans, concentrated in the most recent days. `tj serve` now runs the (idempotent) backfill on startup over a wider window so downtime self-heals, then on an interval so an ongoing live-path miss is closed within minutes. `tj backfill status` and a new `tj doctor` check surface the gap with the time left before Claude Code prunes the source, so it is visible while it is still fixable. Verified live rather than by inspection: booting the daemon against the real corpus ingested 290 sessions / 27,348 spans with no human action and took the gap from 225 to 0; the interval job was observed firing and completing; and repeat passes insert nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The stale-scheme self-heal deletes every backfill-tagged span for a session that is not in the run's keep-set, on the reasoning that only a pre-v0.5.2 uuid-keyed orphan could be missing. That reasoning holds only when the keep-set is the session's COMPLETE span set, and a windowed pass cannot build one: `since` filters files by mtime and by parsed end time, so a session straddling the boundary -- a long conversation whose main transcript is still being appended while its subagents/ files finished days ago -- yields only its in-window files, and the out-of-window siblings' already-ingested spans are deleted as if they were stale. Reproduced deterministically: a two-file session, fully ingested, re-ingested with a two-day window after the subagent file aged out, loses that file's span. On the real corpus a windowed second pass deleted 618 spans. The guard already excluded the quickstart `max_sessions` cap for exactly this reason, and its comment even said the per-session keep-set may be incomplete -- but it named only that one parameter, so the identical hazard one parameter over went unnoticed for as long as `--since` has existed. The predicate is now "is this pass bounded at all", so the next bounding parameter fails safe by construction. Survivable while it only fired on a human-typed command; continuous silent data loss the moment a scheduled job calls the same function every few minutes. A second test pins that an unbounded pass still purges a real orphan, so the guard has not simply disabled the self-heal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
[ingest] config (auto_catch_up, interval_minutes, lookback_hours, startup_lookback_days) drives a `tj serve` startup pass plus a recurring one, alongside the existing retention/relearn/cost-proposal jobs and following the same discipline: its own DB connection per run, on its own thread, so it never touches the request path or delays the bind. The startup window is deliberately the wider of the two, because it has to cover however long the daemon was off while the steady-state pass only covers one interval. A schedule alone is not enough, though: if the daemon is not running, nothing runs. So the gap is also visible without it. `tj backfill status` reports on-disk sessions against ingested ones with the days remaining before Claude Code prunes the oldest missing transcript, and `tj doctor` grows a check for the same thing. The existing live-span freshness check catches "nothing is arriving at all"; it cannot see the steady-state case where most sessions arrive and a slice silently does not. Both compare on the transcript's internal sessionId and report sessions with no assistant turn separately, so a healthy install does not read as permanently broken over sessions that have no usage to ingest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings in the avoidable-vs-observed cost relabelling so the transcript catch-up work merges cleanly on top of it. Disjoint areas: the ingest/daemon layer here, the analyzer/cost-proposal layer there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lead with what was avoidable, not with what re-sending context cost
…contract relearn set observed_cost_usd on its aggregate Review-inbox card but left coverage_note empty, so the card showed a cost figure with no avoidable figure beside it and no explanation why -- inviting the same "the gap must be unavoidable" misread PR #598 fixed for resend. _relearn_to_proposals now states the Rule 27 disjointness against resend's claim and breaks the gated clusters down by reason (no fix template / net-negative / budget-deferred). Also enforces the module's own stated contract ("coverage_note required whenever observed_cost_usd is set") with a dedicated test that checks every priced proposal directly, and states the missing-avoidable-figure case in words on the card instead of a bare dash. Co-Authored-By: Claude <noreply@anthropic.com>
… not measurable yet) `relearn` prices failure episodes -- individual erroring tool calls -- and the standing hypothesis was that this understates the waste badly, because the real cost is the inflated SESSION around the failure: an agent that does not know a project's constraints explores, backtracks, and re-sends a growing context on every turn of that flailing. The proposed replacement was to cluster sessions doing the same repeated work and compare cost across the point where the lesson was written down. This measures whether that is possible before anything is built on it. Four things had to hold; three do not. Repeat-task clusters ARE identifiable -- 346 clusters over 2,711 sessions, with 31 clusters at n>=5 covering 89% of them -- but only via the session's first user prompt, which lives in the agent harness's on-disk transcript. Measured availability of that transcript: 99.9% under 30 days, 3.6% at 30-60 days, 0.0% beyond. The UNINFORMED side of any before/after is by definition the old side, so the high-confidence clusterer can only see the informed half. The span-only fallback -- the only signal that survives transcript rotation -- was scored against the task-statement ground truth over 972 sessions and 200k sampled session pairs. Best operating point: precision 0.54 at recall 0.04 against a 0.147 baseline. Roughly half the pairs it calls "the same work" are not, and tightening the predicate plateaus precision while collapsing recall. Tool-call shape is dominated by which tool the agent is holding, not by what task it is doing. Within the tightest cluster obtainable -- one machine-issued prompt template, one project -- per-session cost has CV 0.7-1.2 and a 95% bootstrap interval of +/-46-49% on the median at n~45. The effect being hunted is ~50%, i.e. it sits at the noise floor. Across the whole corpus only 13 (cluster x codification-event) pairs are testable, and their ratios span 0.16x to 3.35x with intervals excluding 1.0 in BOTH directions -- the signature of a confound. The confound is model routing: 5 of the 7 apparently-significant results had the model mix move underneath them. Controlling for it leaves the one project that motivated the whole idea at null or dearer at every event, priced by the gate at $0.00. So `relearn` keeps pricing failure episodes. The session-inflation hypothesis is not refuted; it is unfalsifiable with current instrumentation, and the write-off says exactly what instrumentation would change that. What lands here: * `core/optimize/repeat_task.py` -- the measurement instrument, reusable when the instrumentation improves. Similarity methods carry their MEASURED precision as data, and `measure_codification_delta` refuses to price a cluster below the bar rather than guessing, returns `null` whenever the bootstrap interval spans 1.0, and prices off the conservative end of the interval when it does fire. Deliberately not registered as an analyzer and emits no figure into any report. * `docs/internal/repeat-task-codification-measurement.md` -- the full write-up with every table, the confound, and the instrumentation changes that would make the question answerable. It also settles an open boundary. The `script` analyzer (`analyzers/workflow_restructure.py`) clusters on ordered tool-call signature, which is the same primitive measured at precision 0.54 above -- so that table doubles as an audit of it. That is not a bug in `script`: a script replaces a SHAPE, and shape is the right unit for "could code do this". It is the wrong unit for "was this the same work". The two analyzers stay separate -- `script` detects shape recurrence and emits a script, `relearn` detects failure recurrence and emits a rule -- and `relearn` must not adopt `script`'s signature as a same-work test. Tests: 26 new unit tests; full unit suite 3165 passed, 1 skipped. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
Author
Added: a measurement of whether codifying a lesson makes repeated work cheaper
The hypothesis under test was that
Across the whole corpus only 13 (cluster × codification-event) pairs are testable at all, and their ratios span 0.16× to 3.35× with intervals excluding 1.0 in both directions — a confound, not an effect. The confound is model routing: 5 of the 7 apparently-significant results had the model mix move underneath them. Holding it constant leaves the one project that motivated the idea at null or dearer at every codification event, priced at $0.00. What this does to the write-budget verdict: neither validates nor overturns it. The measurement that would adjudicate it is not obtainable here. Keep the current suppression rule, and stop describing it as empirically settled in either direction — it is a model, and it remains one. Boundary settled as a side effect. What lands
Deliberately untouched: 🤖 shipped by shiploop |
…canonical past_overspend_usd The product carried three near-identical per-analyzer dollar quantities: `past_overspend_usd`, `estimated_recoverable_usd` (the same number, forward-framed), and `estimated_monthly_usd` (that number times a pace ratio). Measured on a real 2,291-session corpus, the first two were byte-identical for 23 of 23 priced proposals, and the paced aggregate was a third value. Three overlapping names for one quantity is a standing trap for any session asked to change how an analyzer computes. `estimated_recoverable_usd`/`_tokens` and `estimated_monthly_usd`/`_tokens` are removed. Every cost analyzer's finding and every `CostProposal` now carries one avoidable figure, `past_overspend_usd`/`_tokens`, observed over the analyzed window; `observed_cost_usd` stays as the separate, genuinely different total-cost quantity on the two analyzers that measure it. Pacing is gone with the fields it fed. `compute_projection_ratio()`, `_with_rollup_projection`, `_effective_ratio`, the `scaling` field and the cached `cost_active_days`/`cost_n_sessions` pace inputs are deleted. `projection.build_projection_basis` survives for its one legitimate consumer, `write_budget`, which needs a per-month horizon to price what a permanent CLAUDE.md rule costs to keep; it is reached only through `_write_budget_basis`, never touches a past-tense figure, and nothing it produces lands on a payload. `estimated_recoverable_rollup` is deleted and `past_overspend_rollup` is the only aggregate, now carrying the `excluded` passthrough. `GET /relearn/cost-proposals` drops its `rollup` key; the `past_overspend` block is what both leading surfaces read. relearn's clusters are no longer folded in on their own monthly basis: relearn reaches the aggregate through its own `cost:relearn` card like any other analyzer, counted once. `_with_past_overspend` no longer moves a number between fields — it stamps the tense-bearing basis strings only, so there is no copy step for a second name to attach to. Adapters set the canonical field directly. Read-time compatibility: a cache written before the collapse migrates its `estimated_recoverable_*` keys onto the canonical field and DROPS the paced twin rather than promoting it, which would revive the exact defect. Two deliberate, documented exceptions, both on `RelearnCluster` (the per-cluster inbox rows, not a `CostProposal`): its `estimated_recoverable_*` is a fix-gated forward claim over an unbounded corpus, a genuinely different quantity excluded from every aggregate by design, and its `estimated_monthly_*` is the last 30-day-basis figure in the product. Neither may be read by a cost surface or copied onto a `CostProposal`. The field contract is written down in the repo CLAUDE.md — the canonical names, what each means, the time basis each is on, the rule that no user-facing surface may render a paced figure, and the two exceptions. The deletion alone would not stop a future session reinventing a second field. Tests updated rather than dropped: the assertions that pinned the removed distinction now pin the collapse itself (no retired field on any proposal, payload or rollup; exactly one aggregate; legacy-cache migration; relearn's null never coerced to zero). Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
Author
Collapsed the three per-analyzer dollar fields to one canonical
|
| # | Check | Result |
|---|---|---|
| 1 | The three fields really were near-identical | PASS — past_overspend_usd == estimated_recoverable_usd for 23/23 priced proposals; rollup.estimated_recoverable_usd == past_overspend.past_overspend_usd == 5981.720134 exactly |
| 2 | The paced aggregate was a third number | PASS (defect reproduced) — projected_usd_30d = 6002.26871 vs observed 5981.720134 |
| 3 | Collapse changes no number | PASS — same-vintage runs: deadweight and downsize (DB-only analyzers) identical to the cent, 0.000%; resend +0.015%, reuse −0.001%, subagent −0.001%, summarize +0.119%, total +0.092% — sliding-window and filesystem drift (summarize scans the live filesystem, so no two runs are bit-identical regardless of this change) |
| 4 | Retired fields gone from every payload | PASS — zero retired keys across 24 proposals; zero retired or paced keys on the aggregate |
| 5 | Exactly one aggregate | PASS — rollup key gone, past_overspend only |
| 6 | relearn's null not silently zeroed | PASS — past_overspend_usd = None (not 0.0), observed_cost_usd = 46.30 |
| 7 | summarize follows the single-field contract |
PASS — one card, one dollar field |
| 8 | No surface renders a paced figure | PASS — every cost card reads "$X avoidable over the last 30 days"; no "est. / mo" anywhere in the Cost-advisories tab. The Recurring-mistakes (relearn cluster) tab still shows it — the documented exception |
| 9 | Both leading surfaces read one server figure | PASS — Dashboard hero and Review inbox headline render the same number, no client-side derivation |
| 10 | No test regression | PASS — identical failure set before and after (98 pre-existing, environment-related: unicode / onboarding / CLI rendering); 3,707 passing |
| 11 | Lint + compile | PASS |
Tests were updated, not dropped: the assertions that pinned the removed distinction now pin the collapse itself — no retired field on any proposal, payload or rollup; exactly one aggregate; legacy-cache migration; relearn's null never coerced to zero.
One flag for review: docs/optimize/{reuse,subagent,verbosity}.md got a mechanical field-name rename (no prose or positioning change) so the analyzer docs don't name fields that no longer exist. docs/policy/overview.md is the separate policy domain and is untouched.
🤖 shipped by shiploop
Measured against a real corpus, the clustering behind reuse's claim has no content signal (every member's prompt-prefix hash is null, so it falls back to bare tool-name sequence), no time window, no prior-failure exclusion, and the large majority of its dollar figure comes from a null-tool-signature catch-all bucket rather than any actual repeated plan. That makes the claim unrefutable for the wrong reason and, for an interactive coding agent, easy for a user to personally disprove. Gate it through the same persona mechanism that already disables cache/cache-recommend/trim/verbosity/script for claude-code. SDK/mixed/unknown personas are untouched — the code and the concept stay in the tree. Co-Authored-By: Claude <noreply@anthropic.com>
anshss
changed the title
anshss
changed the title
…s verdict
A recurring-mistake row rendered exactly one number: the forward
estimated_monthly_* claim, captioned "est. / mo". That figure is gated --
write_budget zeroes it whenever no permanent rule is worth offering. Measured
on a real corpus: 50 of 55 clusters carry no write, so most rows rendered
"$0.00 est./mo" with no reason given, while those same clusters had already
cost $46.62 of observed spend between them. A zero with no explanation reads
as "this was free" -- the exact inversion of what the gate means.
The per-cluster payback_ratio, standing_cost_tokens_per_session and
past_overspend_* were all computed and carried on the dataclass, and no
renderer read any of them.
Both per-cluster surfaces now lead with what the recurrence already cost
(ungated, past tense) and state a forward figure only where a fix is really on
offer, so a suppressed card can never present its gross saving as claimable:
* CLI (_render_relearn): observed cost per row, a one-line gate note with
the payback arithmetic, and a summary naming how many clusters carry no
fix and why.
* Web (RecurringMistakeRow / the expanded card): observed headline, the gate
note inline, a "what this already cost" block with the re-read-tail
overlap called out, and the payback arithmetic beside the suppressed write.
* The collapsed tail summarises on the same basis as the rows inside it,
instead of advertising "$0.00 combined" over tens of dollars of spend.
write_budget gains short_reason(): the same five verdicts as a short label, so
a dense list can render one line instead of a paragraph and the CLI, the API
payload and the inbox row cannot drift into three names for one flag. Stamped
on write and again on read (mirroring proposal_id / advise_only_reason), so a
cache written by the previous build still resolves -- including via an explicit
alias for the pre-hedging net-negative wording, without which every gated row
would degrade to a generic label for one upgrade cycle.
Also: the net-negative verdict is a MODEL, not a measurement, and every string
describing it now says so. It counts what a rule costs to carry and never what
it saves by PREVENTING a flail; the measurement that would settle that returned
no usable signal. The blind spot is stated once in write_budget's module
docstring with a pointer to the measurement writeup. The suppression itself is
unchanged -- there is no evidence it is wrong, only no evidence it is right.
And a stale comment claiming summarize is deliberately not a COST_ANALYZER; it
has been one for some time, and the comment said the opposite of the code two
hundred lines above it.
Co-Authored-By: Claude <noreply@anthropic.com>
…learn carve-out TEMPORARY
The contract stated the rule as an absolute -- "estimated_recoverable_* and
estimated_monthly_* are DELETED" -- and then carved two relearn exceptions
nineteen lines later. Both statements were individually correct and together
they contradict on a top-down read. That mis-fired in practice within an hour
of being written: a session grepped the retired names, found hits in relearn.py
and index.html, concluded the collapse was incomplete, and filed a ticket
against a deliberate documented exception. It had to be withdrawn the same day.
* The prohibition now carries its scope in its own sentence (deleted FROM
CostProposal and every cost-analyzer finding), with the exception flagged
in the same breath rather than three paragraphs down.
* An explicit GREP TRAP callout: the hits are expected, here is where they
legitimately live, and here are the only two permitted consumers, so a
reader can tell a documented exception from a real violation without
tracing the code.
* The carve-out is labelled TEMPORARY with its three-step retirement
sequence, notes that step 1 is now partially done (both per-cluster
surfaces lead with past_overspend_*), and records that its stated
justification is already eroding.
The blockers on finishing step 1 are named by substance rather than by ticket
number: this is a public repo, and a bare #N renders as a link to an unrelated
issue in this repo.
Co-Authored-By: Claude <noreply@anthropic.com>
anshss
changed the title
anshss
changed the title
htm does not preserve the newline between an interpolation and the literal text after it, so the expanded relearn card read "~329.9k tokensover the scanned corpus". Caught in the live UI, not in a test. Co-Authored-By: Claude <noreply@anthropic.com>
…gure
Two defects in one line of report_from_dict's relearn branch, both caught by
CI's type check on this branch:
* past_overspend_tokens is a non-optional int, but the reader passed
d.get(...) straight through, so a payload written before the field existed
rehydrated it as None. Now int(... or 0), like every other count beside it.
* past_overspend_usd was never restored at all. relearn was the ONE reader in
this module restoring the token field without its USD twin -- every other
analyzer's branch restores both -- so a rehydrated report kept relearn's
~16.7M tokens and silently lost the $46 that went with them. The dollar
field is what every leading surface renders.
past_overspend_basis is restored for the same reason. A test round-trips a
finding through report_to_dict/report_from_dict and asserts all three survive,
plus that a legacy payload rehydrates to the declared defaults.
Co-Authored-By: Claude <noreply@anthropic.com>
…line against resend
relearn priced a pothole at a flat 1,500 tokens — an unmeasured guess the code
described as 'roughly one extra assistant turn's overhead'. Measured across
271,442 real calls on a live corpus, one claude-code turn costs ~140k tokens
(836 fresh + 138,087 cache-read + 795 out), because a coding turn re-sends the
whole context. The constant was ~15-20x low, which is why relearn read as noise:
~70% of the Review inbox's cards carrying 0.8% of its money.
The constant was standing in for TWO different quantities. Split them, each on
the basis it earned:
* HEAD — the forced retry. A failed tool call makes the model emit a recovery
turn a successful call would not have needed. Now measured per cluster from
the sessions it actually occurred in (median billed cost per call, divided
back through those sessions' own input rate; falls back to the prompt's
input-token equivalents when a corpus records no cost). Floored at the text
constant — a retry carries the error text, so it cannot cost less.
* TAIL — the error TEXT re-read on later calls. ~1,500 tokens really is the
right size for a block of error text, so the constant stays where it was
earned.
THE LINE BETWEEN THE TWO ANALYZERS, now encoded rather than left to prose:
resend prices redundant context inside calls that HAD to happen; relearn prices
a call that should never have happened at all. So relearn CLAIMS the head only.
The tail is re-sent context resend already prices in full, so it stays in the
observed figure, broken out as past_reread_*, and is claimed by resend alone.
A test asserts claim + tail == observed, so neither can drift.
Also fixes an invariant this change would otherwise have broken silently: the
below-threshold residue documents itself as being 'on the same head-term basis
as past_overspend_tokens' and was still multiplying by the text constant. It
now moves with the head, and its test asserts the invariant instead of pinning
the old literal.
Measured effect on a real corpus (55 clusters, unchanged — no new detection):
observed $46.45 -> $186.32 ($32.21 tail disclosed, $146.23 claimed)
residue $5.16 -> $86.30
total $51.61 -> $272.62 (5.3x)
The cross-analyzer rollup is unaffected: relearn's CostProposal carries
past_overspend_usd=None by design and reports observed_cost_* only, so it
contributes $0.00 to the headline and cannot double-count against resend there.
Co-Authored-By: Claude <noreply@anthropic.com>
… floor
Three founder calls from reviewing the page live.
1. THE FULL-WIDTH BAND IS GONE, from both the Dashboard and the Review inbox.
It led each page with a large figure, three paragraphs of prose and a
derivation accordion, and read as heavy. The figure now occupies the
compact tile slot the old forward-looking 'est. savings' tile used to hold,
beside 'Fixes applied' — which is the shape the page was asked for.
What did NOT change, because it is honesty discipline and not layout: the
number is still the server's own past_overspend block read verbatim (never
reduced over rendered cards); it is still past tense, observed, and carries
no recovery vocabulary; the total observed COST is still disclosed and still
worded as cost rather than waste (rule 30). The long-form derivation moved
into the figure's hover text rather than being dropped. The excluded-waste
cross-reference moved BELOW the tile row: it used to hang off the band's
note slot and would have disappeared with it, and leaving it unstated reads
as 'the biggest figure the product computes doesn't exist'.
2. A $5 NOISE FLOOR on both tabs. A card worth less than that is not worth a
row — acting on it costs more attention than the fix returns. Measured on a
real corpus the effect is the whole complaint: 51 recurring mistakes -> 5,
15 cost advisories -> 5.
Two rules keep it a display filter and not a quiet write-off. ONLY A KNOWN
FIGURE CAN BE JUDGED SMALL — an unpriced item is not a cheap one, so the
floor never hides it. And WHAT IS HIDDEN IS STATED: each tab renders a line
naming how many rows went and what they cost combined ('46 smaller
recurring mistakes under $5.00 are hidden'). The headline is untouched and
still counts them.
3. The floor is declared ABOVE the selection helpers so the tab count, 'select
all' and 'dismiss checked' all operate on exactly the rows a reader can
see. A bulk action that can reach a row nobody was shown is the failure
mode that ordering exists to prevent. An analyzer-focused deep link skips
the floor, or an Optimize page's 'review it ->' would land on an empty tab.
Tests moved with the decisions rather than being deleted: the band tests now
target the tile and additionally assert it sits IN the tile row, that the cost
disclosure survived the shrink, and that the excluded note is not nested inside
the tile that can disappear.
Co-Authored-By: Claude <noreply@anthropic.com>
This was referenced Jul 26, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A figure may wear the "waste" label only to the extent it was avoidable. Unavoidable or un-analysed spend is cost, and cost is a different claim.
Every change here is that principle applied, or a measurement gap it exposed.
The dashboard used to lead with $1,152.46 "estimated recoverable / mo". That number was wrong in three separate ways at once: it was a forward projection built from a pacing multiplier, it omitted the two largest findings the product computes, and one of its cards was pricing work nobody could have avoided. This PR replaces it with what was actually observed.
What a user sees now
summarizedownsizerelearnreuseMeasured on a real 2,291-session corpus, 30-day window,
claude-codepersona.The changes
1. Lead with what was avoidable, past tense
Both surfaces now render one figure per analyzer:
past_overspend_usd, observed over the analyzed window, never paced. The word "recoverable" no longer appears in user-facing copy.Where an analyzer also knows the total cost of the behaviour, that lands on separate
observed_cost_*fields with cost-only wording and its own rollup key. It is never added to the headline, because the avoidable figure is a subset of the cost — adding them double-counts, and swapping them relabels un-analysed cost as waste.2.
resend: stop implying 94% was unavoidableresendreported $6,972.80 cost of re-sending context beside $398.41 avoidable. Nobody wrote the claim down, but the pairing made it: that 94.3% had been shown to be unavoidable.It had not. The two figures were computed over different populations. Decomposed:
downsize— analysed on another cardMIN_SESSION_CONTEXT_TOKENS, dropped before the calculation ranThree ways of not being analysed. Zero findings of necessity.
context_resendnow partitions its cost by the same predicate the avoidable figure uses (_coverage_class), so the coverage a card states cannot disagree with the coverage the code applied; a test asserts the three buckets re-sum to the total. Acoverage_notestates the split in prose and ends with the load-bearing sentence: the difference between the two figures is what this analyzer did not analyse, not what was unavoidable.3.
downsize: ask about the model's ROLE, not its pricedownsizecontributed $4.80 of a $1,191 rollup. The reflex diagnosis was that its gate —input < 5,000 AND output < 500 AND tool_calls <= 5, summed per session — was unreachable for a coding-agent session.Measured, that was wrong. 231 sessions (10.23%) clear the gate; they carry $6.74 (0.082%) of spend. The gate fires fine and selects sessions worth nothing. Loosening it would have produced more near-worthless cards and moved the dollars by nothing, so the thresholds are unchanged.
The question worth asking was different: a premium model acting as the driver of a long session, doing reads and edits inline instead of routing them to cheap workers. Every inline turn re-reads the accumulated context at the premium rate, so the waste grows with session size — and a gate that sums per session becomes less likely to fire the worse the problem gets.
The new primary case flags a premium-tier driver, no subagent ever dispatched, context past 50K, ≥10 main-thread tool calls, and prices the inline-vs-routed counterfactual in two halves (offload tail, tier delta). 706 of 2,291 sessions. $512.23 of the $516.14 total.
4.
summarize: the second-largest finding, previously on no surfacesummarizewas excluded fromCOST_ANALYZERSbecause it owns its own curate/diff/apply flow. That is a real distinction — and a reason about where the button goes, not a reason to hide the finding.The Review inbox is the complete index of everything actionable, not the list of things whose apply flow happens to live there.
summarizenow produces a normal card carrying its figure into the headline, with the action routing totj summarizerather than offering a misleading inline apply.Left as-is, this omitted the second-biggest number the product computes.
5.
relearn: give it tokenjam's horizon, and stop gates zeroing observed costTwo defects:
It read the wrong source.
relearnis the analyzer premised on long-horizon recurrence, and it scanned Claude Code's on-disk transcripts — which Claude itself rotates at ~30 days. Its "unbounded history" was 37.4 days, while tokenjam's own store held 120. It now reads the DB, so its horizon is what tokenjam retained rather than what Claude has not yet deleted.Three gates turned real cost into $0. Clusters with no fix template (29 of 55), net-negative write budget (17), and budget-deferred (4) all reported zero. But "we have no action for this" is not "this was unavoidable." All 55 clusters now carry an observed cost; the fix-availability question is answered separately from the did-this-cost-money question.
relearnis also adapted to aCostProposal, so it reaches the rollup at all — counted once, through its own card.6. Ingestion was not continuous, and had been failing for a week
There was no watcher, no polling loop, no scheduled backfill, and no catch-up on daemon start. Two paths existed: live OTLP (only if the shell had the env vars and the daemon was reachable at that exact endpoint — no retry, no buffer), and
ingest_claude_code, callable only by a human running a CLI command.Ingested sessions had collapsed from ~200/day mid-month to single digits. Every figure the product showed was a floor.
Backfill was already idempotent — span ids are content-derived hashes and each run anti-joins — so it is now scheduled as a recurring daemon job with a catch-up on startup, and the remaining gap is surfaced to the user instead of being silent.
7.
reuse: gated off forclaude-codereuseclaims two pieces of planning were interchangeable. Unlike every other analyzer's claim, that is not observable. Measured:prompt_prefix_hashwasNonefor all 9 clusters$30.84 of a $5,330 rollup, and the one card a user could personally refute — which would cost them their trust in the $3,968 and $6,972 figures that are sound. The code stays; only the persona gating changes. Full record and the bar for re-enabling:
.claude/product-state/REUSE-ANALYZER-INVESTIGATION.md.8. One canonical dollar field
Three near-identical per-analyzer quantities coexisted:
past_overspend_usd,estimated_recoverable_usd, andestimated_monthly_usd(the same number × a pacing ratio, measured 1.0714).This ambiguity produced a wrong analysis while this PR was being built — a comparison of
past_overspend_usdagainstestimated_recoverable_usdconcluded the two were identical for 6 of 7 analyzers, while the Review inbox was rendering the third value.estimated_recoverable_*andestimated_monthly_*are removed fromCostProposaland every cost-analyzer finding. The contract is written intoCLAUDE.mdas an architectural rule with its rationale, and a test asserts no proposal carries a second per-analyzer dollar field. One documented exception remains onRelearnClusterfor the Applied-tab snapshot, unreadable by any cost surface.9. A measurement that came back negative, reported as such
Does codifying a lesson make repeated work cheaper?
repeat_task.pymeasures it: 13 before/after pairs, ratios 0.16×–3.35×, confidence intervals excluding 1.0 in both directions, and the known-positive project null at all 4 codification events once model choice was controlled for. Priced $0.00.The effect is not disproven — it is confounded, and the telemetry cannot currently separate it. The machinery ships, prices nothing, and says so. Closing that gap needs a masked task-statement hash and the routed model persisted at ingest.
Verification
coverage_noteis now required wheneverobserved_cost_usdis set, enforced by a test rather than a docstring — the same defect had appeared twice.Notes for review
resend'soffloadable_share(0.4021) is still measured across the 175 of 2,291 sessions (7.6%) that delegate at all, then applied to the 92.4% that never do. A structural per-tool-call measure would be better; it is computed nowhere in this tree, so the honest move was to disclose the sample and spread in the basis string. Tracked separately.downsize's card does not yet disclose that $1,787.15 ofresend's cost figure is the same sessions under a different heading. Rule 27 keeps the avoidable figures disjoint; the cost figures still overlap silently. Tracked separately.