gemma-4-E2B-it: current default cq4.zip (2026-07-09 "calibrated") collapses on multi-step reasoning — GSM8K 3/100 vs 87/100 for the previous build, at identical speed

[john-rocky]

Summary

The current default download for google/gemma-4-E2B-it — Cactus-Compute/gemma-4-E2B-it gemma-4-e2b-it-cq4.zip, the file cactus run/cactus download resolve at --bits 4 — appears to have lost multi-step reasoning in the 2026-07-09 artifact swap. The build it replaced (now gemma-4-e2b-it-cq4-uncalibrated.zip in the same repo) does not have the problem, and the two are speed-identical, so this looks like an unintended artifact regression rather than a speed/quality trade.

Measurements

GSM8K, first 100 test questions, 0-shot CoT, greedy (temperature 0, top_k 1), max_tokens 1024, driven through cactus_complete (Metal backend, auto_handoff: false), same prompt + answer extraction for every build:

bundle (sha256 prefix)	GSM8K
cq4.zip (current default, 7a2579ad1089f626…)	3/100
cq4-uncalibrated.zip (7c71f075554bd27b…)	87/100
cq3.26.zip (n=25 probe)	1/25
cq2.54.zip (n=25 probe)	4/25

Supporting observations:

• The two cq4 builds decode at the same speed on an iPhone 17 Pro (50.2 vs 50.6 tok/s, median of 3 cold runs each) — the difference is purely output quality.
• The default build's failures are visible even on single-step questions: in a 9-item sanity prompt it answers 7×8 with "119" and "thank you in French" with "Bonjour"; the uncalibrated build gets 8/8 of its emitted items right.
• On multi-step problems the default build hedges without committing to a reading until the token budget runs out (~4.4k chars mean), the same signature across all 100 questions.
• Your docs (docs/cactus_quants.md) report GSM8K 71.20 for CQ4 and 74.00 for the production bundle — consistent with what we measure on the uncalibrated lineage, not with the current default.
• Timeline from the HF repo history: 2026-07-09 "Add locally-built calibrated CQ4 (GPTQ language, FP16 towers)" + renames made it the default; the 07-16/17 probe additions kept those graphs ("Restore original cq4 graphs; keep cloud-handoff probe").

Repro

Everything is public in https://github.com/john-rocky/apple-silicon-llm-bench:

• harness: scripts/parity_gsm8k.py (--which cactus, drives your Python FFI; pinned question set in evaldata/gsm8k_test.jsonl)
• per-question reports: results/quality/gsm8k_cactus-*.json
• session audit trail incl. device runs: results/raw/2026-07-20-cactus-arm/SUMMARY.txt

python3 scripts/parity_gsm8k.py --which cactus \
  --bundle <unzipped gemma-4-e2b-it-cq4> --cactus-repo <cactus clone> \
  --n 100 --max-tokens 1024

Engine commit 1ace6d78; results are deterministic under greedy decoding.

If the calibrated swap is intentional, a docs note would help users pick; if not, re-pointing the default at the uncalibrated graphs looks like the smallest fix. Happy to re-run the harness against a fixed artifact.