[case] domux-seniorsafe - CPU-only benchmark: normalization pre-pass lifts exact match 55.7% to 85.7% (audit-v2) + frozen generalization audit - #29
Open
qrx-joe wants to merge 9 commits into
Conversation
|
|
Signed-off-by: Lumi Ruvenne <174945156+qrx-joe@users.noreply.github.com>
Signed-off-by: Lumi Ruvenne <174945156+qrx-joe@users.noreply.github.com>
Download Domux at pinned revision 6c71a32f and run all 80 SeniorSafe samples on a CPU-only Windows PC (BF16, greedy), raw and normalized pipelines, zero runtime errors. Normalization pre-pass lifts exact match from 55.7% to 75.7% with a 17.9% regression rate on self-corrections. Commit artifacts, case README, and add the pillow and torchvision CPU deps Gemma4Processor requires. Signed-off-by: Lumi Ruvenne <174945156+qrx-joe@users.noreply.github.com>
…ests Review of the CPU benchmark artifacts surfaced three issues; the committed run outputs and metrics are left untouched as the historical record of the first run. - normalize.py: pad every substitution with spaces so translated tokens stay separate words (把客厅灯设为蓝色 -> "Living Room Light set to Blue", not "Living RoomLightset toBlue", which made the model echo bogus device slots like "Lightset"/"Heaterset"), and add missing lexicon entries 厨房->Kitchen, 三十度->30 Celsius, 安防->Security System. 6 of the 7 normalized-run regressions trace to these splicing defects on clean text, not to the self-correction rule. - README.md: rewrite the safety section to match the artifacts — on all 10 risky/ambiguous samples the model emitted well-formed executable commands in both pipelines; only the rule layer (5 clarify / 5 reject) prevented execution, so the 100% safety metrics are rule-layer-only and true by construction. Also fix the evaluable-sample wording and the regression root-cause analysis. - test_seniorsafe.py: make the run_transformers_cpu import optional so the suite runs on a stdlib-only machine (the CPU-runner test skips without torch), and add regression tests for the spacing and lexicon fixes. - artifacts/discussion_draft.md: sync the pending HF post with the corrected safety and root-cause wording. Signed-off-by: Lumi Ruvenne <174945156+qrx-joe@users.noreply.github.com>
…d preview - Replace PENDING channel placeholders with the published Discussion URL (frontmatter + body), required by scripts/validate_cases.py - Add preview.png benchmark summary card rendered from committed artifacts (metrics.json + real run log excerpt) - validate_cases.py passes; no model weights committed Signed-off-by: Lumi Ruvenne <174945156+qrx-joe@users.noreply.github.com>
qrx-joe
force-pushed
the
case/domux-seniorsafe
branch
from
August 30, 2026 11:54
1dada6f to
c906b98
Compare
…ers, CI tests - Shared wire format (protocol.py): non-empty fields, known actions; exact match preserves order and duplicates; slot/intent F1 via ordered DP - run_support.py: provenance fingerprints (code/data/settings/outputs), exclusive run files, --resume prefix checks, output policy labels (candidate/clarify/reject; execution_performed always false) - normalize.py: space-padded substitutions, missing lexicon entries, correction-context preservation, word-bounded English ASR aliases - score.py recomputes correctness from raw outputs; rejects mismatched versions, duplicate ids, mixed revisions/parameters - CI: run data validation + offline unit suite (torch-free) on PRs Signed-off-by: Lumi Ruvenne <174945156+qrx-joe@users.noreply.github.com>
80 raw + 80 normalized inferences, zero runtime errors, same pinned revision (6c71a32f), same machine, BF16 CPU, greedy, 16 threads. - Normalized exact match 75.7% -> 85.7% (60/70); raw 55.7% unchanged and byte-identical per sample to the first run (80/80) - v2 scorer: slot F1 0.9666, intent F1 0.8652; recovery 23/31 (74.2%), regression 2/39 (5.1%) - REVIEW.md documents scope, remaining failures, and non-goals - verification.json: 31 tests, validators, digest consistency checks Signed-off-by: Lumi Ruvenne <174945156+qrx-joe@users.noreply.github.com>
- Add audit-fix summary and verified rerun section (85.7% normalized); original results kept as clearly-labeled historical evidence - Correct clarify/reject split to 6/4 for the 10 excluded samples (11 clarify / 4 reject across all 80 labels); P95 latency disclosed - Mark discussion_draft.md as the historical pre-fix draft Signed-off-by: Lumi Ruvenne <174945156+qrx-joe@users.noreply.github.com>
…ferences Predeclared, frozen follow-up to the published SeniorSafe benchmark. The protocol was frozen before inference (freeze.json + frozen_sources.zip with data/code/settings fingerprints); predictions and primary evidence were never rewritten afterward. - data/challenge-v1.jsonl + challenge-v1.spec.json: 160 texts / 80 paired scenarios (new values, rooms, device variants, paraphrases, self-correction, negation, repetition, multi-action, ambiguous/high-risk) under a strict input contract; validate_data.py now enforces the spec (--data-spec) and both runners verify frozen files and settings before model access (--freeze). - scripts: generate_challenge.py (fixtures, never model-labeled), freeze_challenge.py (write-once freeze), challenge_report.py (coverage + strict paired scoring), label_sensitivity.py (label-only re-scoring), plus test_challenge.py, test_fixture_snapshot.py and extended test_workflow.py. - CI: validate challenge-v1.jsonl against its spec in cases-lint.yml. - artifacts/generalization-v1: 320 real CPU inferences (zero runtime errors, same pinned revision 6c71a32f), frozen-label exact match raw 75/136 (55.1%) vs normalized 84/136 (61.8%), protocol-label sensitivity 56.6%/64.7%, gate findings (5/24 input-policy false allows, 2/24 blocked-label output candidates, 8 EM regressions), evidence bundle and verification.json; the REVIEW.md states honestly that this is not a blind test and that rewrite bugs (65 percent -> 60, Balcony Light -> Desk Lamp) were not repaired within the frozen experiment — the case is not safe for direct device use. - README: prospective generalization section; the original 80-record narrative stays as clearly-labeled historical evidence. Signed-off-by: Lumi Ruvenne <174945156+qrx-joe@users.noreply.github.com>
qrx-joe
changed the title
Member
|
Reviewed against the HER Hack-Astron #4 acceptance criteria (comment only — winner selection stays with @FenjuFu). Verified
One thing for a maintainer to weigh in on before merge Strong, well-documented entry. |
Member
|
案例本身按 #20 的验收标准有效；合并前请移除本 PR 对 .github/workflows/cases-lint.yml 增加的 SeniorSafe 专属执行步骤。 当前改动会让仓库的通用案例校验工作流在此后每个相关 PR 和 main push 中，永久执行某一份投稿自带的 validate_data.py 与 test_*.py。这扩大了通用 CI 的代码执行面，也把赛事案例的专属测试变成了所有后续案例的固定依赖，不适合作为案例收录的一部分。 cases/domux-seniorsafe/ 下的测试脚本和 README/RUNBOOK 中的本地复现命令可以保留；根 .gitignore 为公开合成数据和证据增加的必要例外也可以保留。移除上述 workflow 专属步骤后可再次复审。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue(s) this PR fixes:
Ref #20 (HER Hack-Astron #4 case entry — referencing, not closing)
What this PR adds
New community case
cases/domux-seniorsafe/: Domux (BF16, full snapshot) running entirely on a no-GPU Windows PC (Intel 14-core CPU, 31.4 GB RAM), benchmarked on an 80-sample synthetic SeniorSafe intent set — 40 clean + 40 paired noisy variants covering ASR errors, dialect-flavored phrasing, code-switching, negation, self-correction, repetition, ambiguous reference, and high-risk ambiguity.Two pipelines are compared: raw utterance vs an auditable rule-based normalization pre-pass:
The original run's 7 normalizer regressions were diagnosed (6 splicing/lexicon defects on clean text, 1 self-correction context drop), fixed in
normalize.py(space-padded substitutions, missing 厨房/三十度/安防 lexicon entries), and verified by the full CPU rerun above. Original evidence stays as clearly-labeled history;artifacts/audit-v2/holds the rerun outputs, rescored history, and verification digests.Generalization v1 (frozen, predeclared)
A later AI-authored challenge froze 160 new texts / 80 paired scenarios (spec + fingerprints) before inference, then ran 320 real CPU inferences at the same revision (zero runtime errors). Strict ordered exact match on the 136 evaluable samples: raw 55.1% (75/136), normalized 61.8% (84/136); protocol-label sensitivity 56.6% / 64.7%. The review honestly documents what the expansion exposes: 8 exact-match regressions, 5/24 input-policy false allows, 2/24 blocked-label requests still marked output candidates, and unrepaired rewrite bugs (e.g. "65 percent" partially rewritten as 60). The case is not safe for direct device integration. Details in
artifacts/generalization-v1/REVIEW.md; reproduction inRUNBOOK.md.Safety: all 10 ambiguous/high-risk samples in the original run route to the deterministic
clarify/rejectlayer; the case states explicitly that the model itself emits well-formed executable commands on those inputs, so the rule gate is mandatory in deployment. No devices were operated at any point.Included: reproduction scripts (dataset + challenge generators, spec-driven validator, normalizer + safety layer, strict v2 protocol / ordered scorer, CPU runner with frozen-run guards, API runner, scorer, validators, tests), both synthetic datasets (80 + 160 texts; no private data; repository license), per-sample raw/normalized outputs for all three stages, redacted environment records, freeze/evidence bundles, and a
preview.pngevidence card.Publication
6c71a32f4d624cadfd9fce9d10240d8068e53456(full BF16 safetensors snapshot, 10,279,032,574 bytes, downloaded from Hugging Face after accepting Gemma terms; no weights committed)Verification
python scripts/validate_cases.pypasses (case metadata, channels, revision format)validate_data.pyvalidates bothseniorsafe.jsonlandchallenge-v1.jsonl --spec challenge-v1.spec.jsonscripts/score.pymatch exactlySpecial notes for reviewers:
The reported safety numbers measure the deterministic rule layer against dataset labels, not model behavior — the model itself emits executable commands on risky inputs and must stay behind the rule gate in deployment. Latency is wall-clock
model.generatetime, no warm-up, single run, CPU-only. The generalization experiment is prospective and synthetic — not a third-party blind test or real elderly speech evaluation — and its rewrite bugs were deliberately left unrepaired within the frozen experiment.