Clean-room LLM vs. identity-resolved data — a customer-insights test

This repo is the companion to our write-up on whether a large language model can replace a consumer-data platform for customer insights. It contains the actual, auditable environment and code we used — so you can inspect exactly how the test was run, and reproduce it on your own data.

No customer data or ground truth is included (see What's not here). Bring your own list, or run against the tiny synthetic sample in data/.

The question

Given a company's own first-party customer list, how accurately can an LLM infer per-person demographics (age, gender) compared to identity-resolved enrichment — and does it need third-party data at all? We scored both, row by row, against attributes the business could verify (age from date of birth; gender), held out of both arms.

The headline result

• Gender is guessable. From public data (SSA first-name statistics) alone, the LLM matched identity-resolved accuracy (~98%) at full coverage. Third-party data adds little.
• Age is not. Inferring a specific person's age from name + ZIP tops out around a ~7–11-year average error (it regresses toward the population mean); identity-resolved age was off by ~2. Even giving the model the domain knowledge a client has ("our base skews older") — validated on a fresh holdout — closed only part of the gap.
• Cost. Because the model wrote a deterministic script rather than reasoning per row, the whole run cost ~$1.23 once, then ~$0/person at any scale. Naive per-row LLM processing would run ~$10k–$30k per million records, and recurring.

The line: the aggregate is guessable; the individual is knowable only from resolved data.

The clean room (why the test is fair)

The single biggest risk is the model cheating — looking a person up online and retrieving the answer instead of inferring it. We made that impossible by construction:

• cleanroom/docker-compose.yml — two containers, two networks. The workbench runs on an internal: true network with no route to the internet; its only path out is a proxy.
• cleanroom/squid.conf — the proxy is an allow-list: it permits HTTPS to the model API and api.census.gov, and denies everything else. People-search sites, general web, and any internal endpoints are unreachable.
• The workbench mounts only the input list (read-only) and public SSA data. No proprietary code, credentials, or graph.
• Every request that leaves the box is written to out/proxy-logs/access.log — so you can prove what was and wasn't contacted.

"No cheating" isn't a promise — it's a config file and a log you can read in a few minutes.

Layout

cleanroom/   docker-compose.yml, squid.conf, workbench.Dockerfile, fetch_ssa_names.sh, cost_summary.sh
pipeline/    infer.py (the model's inference pipeline), infer_variants.py (age-prior variants)
eval/        eval_harness.py, compare_variants.py (scoring), rowbyrow_cost_model.py (cost model)
prompts/     01_kickoff.md, 02_prior_exploration.md, 03_holdout_run.md (the exact operator prompts)
docs/        RUNBOOK.md, INSTRUMENTATION.md, operator_brief.md
data/        arm_input_sample.csv (synthetic; replace with your own list)

Reproduce it

1. Get keys: a model API key, and a free Census key from api.census.gov/data/key_signup.html.
2. Load public SSA names: bash cleanroom/fetch_ssa_names.sh (downloads the SSA national baby-names dataset into data/ssa_names/).
3. Bring a list: put your customers in arm_input.csv (see data/arm_input_sample.csv for the columns), or use the sample.
4. Run the clean room: docker compose -f cleanroom/docker-compose.yml build && ... up -d proxy && ... run --rm workbench bash. Verify the boundary (see docs/RUNBOOK.md), then drive the model with the prompts in prompts/.
5. Score: hold out true age/gender and run eval/eval_harness.py / eval/compare_variants.py.

See docs/RUNBOOK.md for the full step-by-step.

What's not here

Deliberately excluded, to protect privacy and keep this reproducible-by-anyone:

• No customer data — no real input list, predictions, ground truth, or model transcript (all contain personal information).
• No API keys.
• No vendor-internal extraction code — the piece that pulled our specific list from our warehouse is omitted; export your own list to arm_input.csv instead.

License / status

Research artifact accompanying the write-up. Token counts in the cost model are estimates; accuracy figures are from a single pilot (one client, small sample) — directional, not gospel. See the write-up for full caveats.