feat: add deterministic visual-change detection for PRs.

[BillLeoutsakosvl346]

Summary

Add deterministic visual-change qualification for staging PRs across product, landing pages, emails, documentation, desktop, shared components, themes and assets. The engine reads immutable Git objects and compares merge-base to head. Decisions remain flag/exempt; incomplete static evidence flags when a changed input is connected to rendering. Completed analyses exit 0; operational failures exit nonzero with flagged: null.

Schema 3.0.0, engine 0.3.0, policy 3.0.0 replace schema 2. Findings group changed sources, retain locations and representative consumers, and mark usage counts as partial. Full semantic hashes are compared before 4 KiB value previews and deterministic sampling within a 5 MiB report cap.

The resolver narrows imports, selected properties, environment fields and supported helper returns; distinguishes SQL/telemetry from rendering; handles supported equivalent refactors; traces rendered state setters and captured title maps; reads configured Fumadocs OpenAPI inputs; and compares rendering dependency closures instead of every lockfile edit. All in-scope changed files are analyzed. After qualification and a nearby unchanged rendering-consumer examination, additional unchanged consumers may be omitted with an explicit coverage limitation. This is a PR qualifier, not an exhaustive inventory of indirect effects.

The advisory production workflow uses trusted default-branch engine/configuration, pinned tooling and read-only permissions, and uploads only a JSON artifact with seven-day retention. Activation requires reaching main; this draft PR does not activate the production trigger. No application code/plugins execute during analysis. No PR findings comments, labels, annotations, required-check registration, Slack, screenshot or AI stages are added.

See scripts/design-diff/README.md and the committed frozen manifest/replay tooling under scripts/design-diff/benchmark/.

Type of Change

• Bug fix
• New feature
• Breaking change (draft JSON schema 3 replaces schema 2)
• Documentation
• Other

Testing

• Bun 1.4.1: 486 script tests across 33 suites pass. Engine types, script-test discovery, targeted Biome, required API-validation audit, Actionlint and Git whitespace checks pass. Dependencies were installed frozen with lifecycle scripts disabled.
• Tests cover visual categories, movement, selected bindings, supported equivalent refactors, OpenAPI inputs, Git edge cases, bounded reports, byte-identical repeated/truncated output and source/plugin non-execution sentinels.
• Immutable engine: c8d1bf882d9fbfc73adfb7eb71d1cd46b3e2b286. The formerly missed tool-progress changes feat(browser): add verified form filling and horizontal scroll #7556 and feat(desktop): improve semantic browser tool reliability #7494 now flag with progress-title UI evidence. Repeated feat(browser): add verified form filling and horizontal scroll #7556 reports are byte-identical locally and match the cloud report byte-for-byte.
• All 180 comparisons are accounted for: 176 completed, four timed out. All 95 completed clearly visual cases flagged; two further clearly visual cases timed out.
• Final replay: first six comparisons, 90 preserved result records, and 84 resumed comparisons. The middle run was cancelled after slow queue scheduling; completed records and genuine timeouts were preserved, and only unstarted comparisons were resumed. Engine/configuration identity, frozen commits/file sets/labels, result hashes and cache identities are verified.
• Baseline: 108/120 qualified (90%); 39/49 frozen-nonvisual PRs flagged (79.6%). Revised results below use completed analyses only and disclose failures. Frozen labels are unchanged.

Cohort	Flagged / completed	Qualification among completed	Failed / expected	Frozen nonvisual flagged	Clear visual flagged	Uncertain flagged
original	79/117	67.52%	3/120	11/47 (23.4%)	59/59	9/11
holdout	41/59	69.49%	1/60	3/18 (16.7%)	36/36	2/5

On the same 117 completed original comparisons, qualification falls from 105/117 to 79/117; on the same 47 completed nonvisual cases, flags fall from 37/47 to 11/47.

Original frozen labels contain 60 clearly visual/content cases, 49 nonvisual and 11 uncertain; holdout contains 37, 18 and 5. Holdout labels were frozen before results, but some cases were subsequently used during debugging, so this is not a wholly blind evaluation. Documentation and visible copy are in scope; clear visual cases alone make up 50% of the original sample.

• original: possible full-cohort qualification range 65.83–68.33%; failures are neither clean nor confirmed flags.

• holdout: possible full-cohort qualification range 68.33–70.0%; failures are neither clean nor confirmed flags.

Acceptance remains incomplete. Operational failures: #6593 (Analysis deadline exceeded), #7277 (Analysis deadline exceeded), #7625 (Analysis deadline exceeded), #6901 (Analysis deadline exceeded). Clear completed cases without sufficiently precise reviewed presentation evidence: #7686, #7452.

Manually reviewed relevant presentation evidence is established for 93/97 received clear visual cases. Every frozen-label disagreement is reviewed. #7573 is a documented label correction: MIME normalization changes file-picker disabled states; it stays in the primary frozen-nonvisual count rather than silently improving the rate.

Remaining apparent false positives include broad authentication/environment/provider records and complex refactors (#7037, #7207). Arbitrary runtime behavior, opaque factories, dynamic inputs and event/effect scheduling are not fully modeled. Evidence is not proof of changed pixels; partial usage counts are not counts of visually changed instances.

Cohort	Median / max analysis seconds	Max peak RSS	Max report size
original	80.99 / 477.51	3.56 GiB	4,393,675 bytes
holdout	78.15 / 276.11	2.40 GiB	4,636,406 bytes

Metrics cover completed analyses; timeout peak RSS is unavailable. Each CLI comparison has a 900-second analysis deadline. Batched research jobs have a 45-minute outer deadline for six comparisons; production remains a 15-minute complete job including setup, whose reliability is not established by this benchmark. Every completed report stays below 5 MiB.

Final cloud measurement wall time was 57m 05s, including queue/setup gaps; prior tuning runs are excluded. The last 84 comparisons took 22m 23s after batching.

Latest engine-head PR CI: 14 passing, 25 skipped, none failing or pending when checked. CI success does not override benchmark failures or activate the production workflow.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing locally
• No new warnings introduced across all application CI checks
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Screenshots/Videos

Source-analysis automation; screenshots are a later stage.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 11, 2026 5:50pm UTC