chief-product-trailer.mp4
Your agents don't need more power. They need a chief of staff.
Quickstart · How it works · Evaluated, not asserted · Connect your agent · Docs · 简体中文
24 events in → 1 interruption (96% intercepted: 14 blocked outright, the rest batched, dispatched, or remembered) · only 75% of events ever reach the LLM — the noisiest 25% dies on hard rules in microseconds, for free · stable-prefix prompts: 70% of judge input tokens cache-hit (system + context blocks) · projected judgment cost $0.104 per 1,000 events (DeepSeek list prices, cache-aware)
(every number regenerates from the deterministic demo replay: make readme-metrics)
Chief sits between you and everything that wants your attention — agents, heartbeats, CI, RSS, watchers. Everything flows into it; it thinks for itself; then it does exactly one of three things:
- 🔔 Interrupt you — only when worth it, at the right moment, arriving with a plan.
- 🤖 Dispatch work to your agents — and verify the result before reporting ("done" is a claim, not a proof).
- 📚 Curate into memory — facts and intents not worth mentioning now, waiting to be connected later.
A day of an engineer's life: 24 events in → 14 blocked · 6 batched · 3 handled · interrupted exactly once.
| 🧠 Three-stage worthiness engine | Hard rules (µs) → similarity classifier (ms) → LLM judge (only when needed). Cheap first, smart last. |
| 🎭 Scene-aware timing | Sleeping / deep work / meeting / commute — per-scene interrupt thresholds. The same event rings at your desk and waits in a digest at 2 AM. |
| 🕶️ Shadow mode | For its first 7 days Chief never actually interrupts — it shows you what it would have done, and earns the right to ring. |
| 📝 Explainable, editable policy | Everything it learns distills nightly into a human-readable POLICY.md. Your edits win, effective immediately. |
| ✅ Verified dispatch | Agents report "done"; Chief checks. Acceptance command or LLM second opinion — fails closed. |
| 🔌 Protocol, not pipes | In: one POST /v1/events (or MCP propose, or chief push "…") connects anything in minutes — message your Telegram bot and the verdict comes back to your phone. Out: one signed webhook makes any HTTP receiver a delivery channel. |
| 🔒 Local-first | One SQLite file + markdown under ~/.chief. No hosted service or telemetry; the local console stays on 127.0.0.1. |
| 🔬 Evaluated, not asserted | 418 offline tests, a 200-case golden set, a 100-user learning benchmark, per-decision USD cost. Every claim below ships with a command that proves it. |
uvx agent-chief demo # zero keys, zero config, fully offlinePrefer pip? pip install agent-chief installs the same chief command.
You'll watch a day of an engineer's life replay: 24 events in → 14 blocked · 6 batched · 3 handled (all verified) · interrupted exactly once.
Ready for real sources?
uv tool install agent-chief # install the persistent `chief` command
chief init # 60s wizard, every question skippable
chief run # resident brain + local consoleIf you run heartbeat agents, you know the ritual: "All clear, nothing to report." Every few hours. Forever. Each one costs a glance, and the glances add up until you stop reading — including the one that mattered.
Chief drops zero-information reports on the floor (regex and embedding similarity against a canned empty-report set, both required — a security scan that mentions "all clear" still gets through). The demo opens and closes with this, because it's the single most requested feature among heartbeat users.
No decision is a black box. Every Decision carries its reason, five scored
components, the rules it matched, and what it cost — and chief trace
replays the whole chain after the fact:
$ chief trace evt_20260706_1040_ab12
CI failed on main: test_auth_flow broken by PR #482 dev.ci · github-actions
route dispatch at stage 3 in scene deep_work (confidence 0.85)
score 0.87 urgency=0.90 relevance=0.90 actionability=0.85 novelty=0.80 confidence=0.90
┌────────────┬───────┬──────────────────────┐
│ stage │ ms │ note │
│ stage1 │ 0.1 │ no hard rule fired │
│ associate │ 1.2 │ 0 memory hits │
│ judge │ 812.4 │ backend deepseek │
│ route │ 0.3 │ routed dispatch │
└────────────┴───────┴──────────────────────┘
tokens: 1104 in (704 cached) / 96 out · prompt v1 · cost $0.000301Prompts are versioned templates (judge/templates/v1/), the version is
stamped into every decision, and no prompt change merges without an eval
diff on the 200-case golden set (chief eval --compare v1 v2). If the LLM
backend dies, Chief degrades to rules-only conservative routing — never
interrupts while blind — and heals itself when the backend returns.
The 👍/👎 you give ("worth my attention" / "don't bother me") is a reward signal; the per-topic EMA weights are the policy; feedback trains them. Chief ships an eval that measures the loop closing — a simulated user with hidden preferences, corrected only by the ±1 signal:
$ chief eval --learning
Routing agreement: 0% → 100% (+100%) · converged in 2 rounds
r 0 | | 0%
r 1 |█████████████████ | 86%
r 2 |████████████████████| 100%No labels, no gradient, no black box (SPEC §13: explainable by construction).
It corrects the borderline calls — the ones that actually need judgment,
since stage-1 rules already handle the obvious. Watch the learned per-topic
lean in the console's Learning tab, or reproduce the curve with
chief eval --learning.
One user is an anecdote — so there's a cohort benchmark too.
chief eval --cohort runs the same loop over a committed, seeded 100-user
dataset (eval/personas.jsonl) — each with hidden
preferences, a different scene, and a different feedback-noise level. It's a
proper train/eval split: ±1 feedback trains each user, then interrupt
quality is scored on a held-out event stream they never trained on.
$ chief eval --cohort
cohort 95% of 100 users converge · held-out interrupt F1 0.10 → 0.87
EMA weights alone 64% |███████████ | ← ceilings, provably
+ learned pins 95% |████████████████| ← breaks the ceiling
held-out F1 (mean) 0.10 → 0.87With EMA weights alone, only 64% converge — and the other 36% aren't a
mystery the benchmark hides: they are provably the users with a
wanted-but-too-quiet topic that bounded weights can't lift over their scene's bar
(a topic of strength s peaks at 5s², so it clears threshold T only when
s ≥ √(T/5)). Rather than nudge weights forever at a topic they can't reach,
Chief escalates to a hard per-topic pin once the correction keeps coming but
the weight step has stopped moving — the "you've told me six times, I'll just
always flag this" move. That rescues 31 of the 36 capped users, lifting
convergence to 95%; the 5 who remain are all the noisiest-feedback users, so
the residual ceiling is noise-limited, not arithmetic. Full derivation and
three worked users: docs/eval/cohort-benchmark.md.
Why this matters: most "it learns your preferences" claims are an unfalsifiable demo. This one ships the population distribution, the exact boundary where nudging weights stops working, and the escalation that breaks it — being able to name where your system fails is more convincing than claiming it never does.
📝 The engineering story behind Chief — the funnel, per-model cost accounting, and the falsifiable learning loop — is written up in docs/blog.
Every number on this page is backed by a deterministic, offline eval you can run yourself — no keys, no network. Claims that can't be measured don't ship.
| Eval | What it proves | Run it |
|---|---|---|
| Routing regression | the demo's 24 events route exactly as pinned — CI-gated at 100%, forever | chief eval |
| Capability | agreement on a 200-case golden dataset whose labels are verified against the real pipeline | chief eval |
| Reward loop | ±1 feedback trains the policy — one user, 0% → 100% in 2 rounds | chief eval --learning |
| Cohort benchmark | it generalizes to 100 users: 95% converge (learned pins break the EMA ceiling), held-out interrupt F1 0.10 → 0.87 | chief eval --cohort |
| Preference drift | preferences move and Chief tracks them: held-out F1 0.86 → 0.69 → 0.88 across a mid-stream flip, and 100% of stale pins are dropped | chief eval --drift |
| Ablation | each funnel stage is load-bearing: hard rules save +42% of judge calls & +20 pp; the judge adds +38.5 pp over the rules-only floor | chief eval --ablation |
| Calibration | the score you route on is trustworthy: raw salience ranks backwards (AUC 0.368), learning inverts it to 0.918; isotonic cuts ECE 0.263 → 0.011 | chief eval --calibration |
| Red-team | 16/16 hostile payloads contained — injection can't force an interrupt, malformed input fails closed, the executor stays argv-only (§13) | chief eval --redteam |
| Cost accounting | per-model, cache-aware USD on every decision — this caught a 17× mispricing bug | chief trace <id> |
| Graceful degradation | judge offline → rules-only conservative routing, never interrupts blind, auto-heals | chaos-injection tests |
| Prompt governance | no prompt change merges without an eval diff on the golden set | chief eval --compare v1 v2 |
418 tests, fully offline. The demo routing table is a full-table regression — every event's route is pinned, so a behavior change can never slip through silently.
make test lint # pytest + ruff, no keys needed
make demo # the offline day-of-engineer replay
make release-check # build the wheel, run the demo from it via uvxFor its first 7 days (or 50 graded samples), Chief never actually interrupts
you. Would-be interrupts land in the digest annotated
⚡ would have: interrupted you (score 0.87, scene deep_work) with ✓/✗ grading
buttons. You watch it think, grade its calls, and only when it graduates does it
earn the right to ring. Graduation comes with a Tact Report (chief report).
Two axes, never one: content worthiness × scene tolerance.
flowchart LR
subgraph sources [Anything]
A1[Agents] & A2[CI / GitHub] & A3[RSS / watchers] & A4[Heartbeats]
end
sources -->|"POST /v1/events · MCP propose"| N[Normalize<br/>+ topic inference]
N --> S1{"Stage 1<br/>hard rules (µs)"}
S1 -->|survives| S2{"Stage 2<br/>similarity (ms)"}
S2 -->|unfamiliar| M[Associate<br/>memory] --> S3{"Stage 3<br/>LLM judge"}
S3 --> R["score × scene<br/>threshold"]
S2 -->|familiar| R
R --> I[🔔 interrupt]
R --> D[📰 digest]
R --> P[🤖 dispatch → verify] --> I
R --> C[📚 curate]
R --> X[🗑 drop]
SC["Scene engine<br/>clock · calendar · focus"] -.-> R
L["Learner<br/>EMA · nightly distill"] -.-> S1 & S2 & R
- A three-stage worthiness engine: hard rules (µs) → similarity classifier (ms) → LLM judge (pluggable: Ollama local, DeepSeek, Anthropic, OpenAI).
- A scene engine (clock, calendar, focus, lock state — pluggable providers) with per-scene interrupt thresholds; low-confidence scenes degrade toward silence.
- Every learned preference distills nightly into a human-readable
POLICY.mdyou can read and edit; your edits win, effective immediately.
Deep dive: docs/architecture.md.
export CHIEF_TOKEN="$(chief token)"
curl -X POST http://localhost:8787/v1/events \
-H "Authorization: Bearer $CHIEF_TOKEN" -H "Content-Type: application/json" \
-d '{"source":"my-agent","topic":"dev.ci","summary":"CI failed on main"}'Chief answers with a Decision — route, score, and a one-line reason. MCP agents
use the propose tool instead, and chief lite gives zero-daemon judgment for
one-shot callers. Full contract: docs/protocol.md
· runnable samples: examples/.
chief ui (or the resident chief run) serves a local console at
http://127.0.0.1:8787/ui — single user, token-gated, no cloud:
- Today — digest queue, interrupts, LLM share and spend at a glance
- History — every decision with its reason, score, cost; searchable
- Rules — edit
POLICY.mdin place; your edits win, live immediately - Tasks — approve/reject dispatched work (verification still applies)
- 👍/👎 on every decision — "worth my attention" / "don't bother me"; Chief's learner weighs these above every inferred signal
chief connect composio --secret whsec_… # GitHub, Gmail, Slack, 500+ apps
chief connect github # gh notifications poller
chief connect rss --url https://hnrss.org/frontpage
chief sources # see what's wired upComposio is the flagship connector: point its
trigger webhooks at /v1/connectors/composio (HMAC-verified) and every
connected app flows through Chief's judgment. The connector registry leaves
documented slots for zapier/n8n-style automations and MCP-push agents — one
adapter module each.
Drop-in skills teach agent hosts to route through Chief instead of pinging you:
- skills/claude-code/ — Claude Code agents
propose via
chief lite(no daemon needed) and obey the route. - skills/openclaw/ — OpenClaw agents propose via MCP; interrupts ride OpenClaw's own channels back to you.
Both encode the same iron rule: the agent MUST NOT message the user directly.
Noisy upstream bots in, one accountable judgment layer in the middle: examples/integrations/ ships a runnable stock-analysis-bot feed (watch five "all good" reports die and three real findings survive) and a generic webhook template any agent can copy — both fully offline.
core/ brain loop, 3-stage scorer, learner, digest, SQLite state
context/ scene engine + providers (clock, calendar)
judge/ pluggable LLM judges: ollama · deepseek · anthropic · openai · fixtures
ingest/ webhook (FastAPI), MCP server, GitHub/RSS pollers, normalizer
dispatch/ executors (claude-code, whitelisted shell) + verification
delivery/ terminal · desktop · telegram (feedback buttons)
memory/ curate, associate, expire → archive
demo/ the offline day-of-engineer replay fixture + runner
eval/ golden 200-case dataset, reward-loop + 100-user cohort benchmarks
skills/ OpenClaw integration (propose-and-obey)
See ROADMAP.md for what's deliberately out of v1 (hosted or
multi-user web UI, cloud sync, Slack/Discord delivery, …) and
CONTRIBUTING.md to get
hacking — the dev loop is uv sync --dev && make test. Design decisions live
as one-line ADRs in docs/decisions.md;
SPEC.md is the full implementation spec the project was built from,
step by step (PROGRESS.md).
Local-first by construction: one SQLite file + markdown under ~/.chief.
No hosted service, no telemetry, no arbitrary shell execution. The web console
binds only to 127.0.0.1; other network calls are the ones you configure (your
LLM backend, your Telegram bot).