feat(vision-mixer): shader FX engine — custom GLSL looks, wipes and master FX takes (GPU) - #626
Merged
Conversation
VideoEffect (chroma key, pixelate, blur, duotone, vignette, VHS, old film, edge glow) with serde defaults, parameter clamping and color validation; EffectTarget (input/master) and request/response types. VisionMixerState gains fx_available + current effects; new VisionMixerEffectChanged WS event. DEFAULT_ENABLE_FX block default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Composable GLES2-style fragment library: prelude (glshader's built-in tex/time/width/height uniforms), shared helpers (luma, hash, soft reveal), timing block (u_start/u_duration evaluated against buffer-PTS time — animation lands at composite time with zero per-frame CPU), 8 looks, 8 wipe masks, 6 master envelopes. glshader only compiles its fragment property while it has no shader; runtime swaps require answering the create-shader signal. The attached handler compiles the element's current fragment on the GL thread and keeps the previous shader on compile failure (a runtime swap can never kill the pipeline). Validation tests run every fragment through a real GL pipeline (llvmpipe-compatible) and assert that a runtime swap actually changes rendered pixels — the regression test for the silent-no-op swap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three glshader FX slots in the GPU pipeline (enable_fx, default on):
fx_look_{i} before each input tee (persistent per-source looks that
follow the source everywhere), fx_take_{i} on the dist branch (wipe
masks, PGM only), fx_pgm after the mixer (master looks + take
envelopes).
New shader transitions: wipe_left/right/up/down, clock_wipe,
iris_open/close, blinds, checker_wipe, noise_dissolve, luma_wipe,
ripple (mask on the incoming source + step-off on the outgoing), and
master-FX takes glitch_cut, flash_dissolve, whip_pan_left/right,
punch_zoom, pixelate_take, film_burn (delayed cut/fade/push underneath
a full-frame envelope). All downgrade to Fade without the FX engine;
PiP-aware takes keep master envelopes on top of the fade.
POST /blocks/{id}/effect applies looks (param-only changes swap
uniforms without recompiling); state endpoint reports fx_available +
current effects; VisionMixerEffectChanged broadcasts changes.
End-to-end test drives the GPU pipeline through looks, wipes and
master takes on software GL.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
WIPE and FX transition bases with shape/kind sub-pickers, LOOKS panel with per-source + master effect selectors and live parameter controls (throttled, uniform-only updates on the backend). FX controls are hidden when the engine is unavailable (CPU backend); state syncs via /state and the VisionMixerEffectChanged WS event. Page version 2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oks rows The 12 wipe-shape buttons squeezed into one flex row and overlapped; wipe and master-FX pickers are now 6/4-column grids with uniform buttons. Looks panel rows align on a label/select/params grid with styled controls. Transition panel widened to fit the grid. Page version v2.29-shader-fx-ui. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…the mask Wipe masks render upstream of the mixer while pad alpha applies inside it: when the incoming pad turned opaque, in-flight frames rendered before the shader swap (mask fully open) flashed the destination for a frame or two before the masked frames arrived. Wipes now run inverted on the OUTGOING branch: the incoming source sits underneath at full alpha and the outgoing source's mask eats away on top. Every pre-swap in-flight frame then renders as 'PGM unchanged' — the race is structurally impossible this way around. The take-start neutral reset must also zero u_invert: uniform values persist on the GL program, and a leftover inverted mask at p=1 would turn 'fully revealed' into 'fully transparent' and black out the branch on its next non-wipe take. from==to takes are now no-ops in the shader paths (an inverted self-wipe would wipe PGM to black). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s broke inverted wipes The inverted wipe relies on the outgoing source covering the incoming one. With a letterboxed outgoing source (e.g. 2.39:1 on a 16:9 canvas) the incoming pad at full alpha underneath showed instantly in the uncovered bands — wipes between mixed-aspect sources read as hard switches (observed in production: 1280x534/546 vs 1920x1080 inputs). shader_wipe_take now picks orientation by geometry: - outgoing rect covers incoming -> inverted mask on the outgoing branch (glitch-free by construction), as before - otherwise -> classic mask on the incoming branch above, with pad alpha turned on two-three frames late so pre-swap in-flight buffers (mask fully open) drain unseen, and outgoing remnants outside the incoming rect fading over the wipe (the slide transition's rule) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sources froze wipes The shader's time uniform is the RAW buffer PTS of the branch it runs on, but u_start was the mixer's output position. Local sources happen to share that timebase; SRT/TS sources carry PCR-derived timestamps that sit far from it, so progress clamped to 0/1 and wipes read as hard switches whenever the masked branch was an SRT source (matching the observed pattern: wipes only animated when the masked side was a local test source, and long durations showed only the tail end). The wipe fragment is now installed with parked uniforms (u_start far in the future = p=0: opaque for inverted, hidden for upright) and a self-removing one-shot pad probe latches u_start from the branch's first buffer PTS — correct in every source's own timebase. Mixer-side cleanup steps got a small end grace so they land after the mask completes. The letterbox repro test now drives both orientations through real pixel checks (mid-wipe must show both sources, ends must land on the right one) and persists the PGM/PVW swap between takes the way the API handler does. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…+ tuning New wipes: melt (Doom-style column drip), barn_doors, heart_iris, star_wipe, pinwheel, crosshatch, hex_dissolve, warp_wipe (smeared directional edge). New master FX: zoom_blur (radial streaks), spin, tv_roll (sync-loss roll), negative_flash, and ripple reworked as a master FX — on the PGM slot the expanding ring distorts BOTH pictures through a crossfade (as a wipe mask it could only ripple one branch). New looks: CRT (barrel + scanlines + grille), halftone, thermal, night_vision, posterize, underwater. Tuning from operator review: checker_wipe runs a denser 20x11 grid; noise_dissolve uses smooth two-octave value noise (organic blobs instead of hard grain); glitch_cut/punch_zoom/pixelate_take get wide-topped envelopes and stronger peaks so the midpoint cut hides behind the effect; film_burn removed. All 40 fragments CI-validated through real GL pipelines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hind MORE A transition picker that leads with hearts and pinwheels reads as a prosumer toy to the broadcast engineers evaluating the mixer. The full catalog stays (CI-validated, zero runtime cost) but the default view now shows only the production staples: - Wipe grid: directional arrows, iris open/close, barn doors, luma wipe visible; the 11 novelty shapes (clock, blinds, checker, noise, melt, heart, star, pinwheel, crosshatch, hex, warp) sit behind a MORE toggle. - Master FX grid: flash dissolve visible (and now the default kind, replacing glitch_cut so the default selection is never hidden); the other 10 takes behind MORE. - Looks dropdown: chroma key, blur, pixelate, vignette listed directly; the stylization looks (VHS, CRT, thermal, ...) under a "Stylized" optgroup. MORE expansion persists browser-wide via localStorage (same vm-* pattern as the other UI prefs). Collapsing a grid while a novelty selection is active falls back to the tier default so the picker never shows a state it is hiding. API surface unchanged — all types remain accepted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ded pickers The 250px panel squeezed the six type buttons, and an expanded FX grid pushed the duration row past the controls bar's 200px max-height cap, clipping it (overflow: hidden). - Transition panel min-width 250px -> 340px - FX grid 4 -> 6 columns (expanded FX = 2 rows instead of 3), matching the wipe grid - Controls bar cap 200px -> 280px; it is a cap, not a fixed height, so the bar only grows when a picker is expanded (4 wipe rows = ~270px worst case) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Heart: the implicit heart curve is taller above the origin (lobes ~1.2, tip -1.0), so its visual center sits above y=0 — and the old "nudge toward optical center" shifted the rendered shape the wrong way on top of that, leaving the heart ~0.2 frame-heights too high. Shift the evaluation point up (+0.1) so the shape drops onto the optical center. Warp: the smear was a single-tap 0.12 UV offset confined to the alpha soft band — barely visible. Replace it with a one-sided 9-tap directional blur (same idiom as master_whip) whose strength eases out over a band ~3x wider than the alpha edge, so the picture reads as being dragged along with the sweep. Parked / fully-revealed pixels take an early single-tap exit, so the installed-but-idle shader does not pay for the blur taps. Validated via shader_validation_test (compile + runtime swap). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The looks panel reuses .pip-config-panel, whose 22vh cap (plus the controls bar's own cap) forced a scrollbar after a handful of source rows. Reuse the PiP layout editor's mechanism: an open looks panel sets .editing on itself (panel cap -> 70vh) and fx-panel-open on body (controls cap -> 70vh), so all rows are visible. Both caps snap back when the panel closes; extreme input counts still scroll within 70vh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…takes
A master-FX take used to program its envelope onto fx_pgm, the same
glshader that carries the persistent master look, evicting the look
permanently (documented, but a UX wart and an asymmetry: per-input
looks already survive wipes because inputs have separate look/take
slots).
Mirror the per-input design on the PGM output:
mixer -> fx_pgm (master look) -> fx_pgm_take (take envelope) -> ...
- apply_master_envelope() targets fx_pgm_take and no longer clears
master_effect state
- reset_take_fx() always neutralizes fx_pgm_take and never touches the
look slots (the master_is_look special case is gone)
- e2e test now asserts the vignette look still sits on fx_pgm after a
glitch take, with the envelope on fx_pgm_take
- operator guide + looks-panel hint updated (the "replaced by FX
takes" caveat is gone)
Cost: one extra identity glshader pass on PGM, matching what every
input already pays for its look/take pair.
Tests: vision_mixer_fx_test, shader_validation_test,
pipeline_lifecycle_test all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ullet Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
srperens
force-pushed
the
feat/vision-mixer-shader-fx
branch
from
June 5, 2026 09:58
e617410 to
3f18931
Compare
This was referenced Jun 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A custom-GLSL effects engine for the vision mixer, GPU backend only (block property Shader FX, default on). Previously stacked on #625 (now merged) — rebased onto main, so the PR now contains only the FX work.
Four
glshaderFX slot kinds in the GPU pipelinefx_look_{i}fx_take_{i}fx_pgmfx_pgm_takefx_pgmThe PGM output mirrors the per-input look/take split: looks and takes run on independent slots everywhere, so a master-FX take plays on top of the master look and the look stays on afterwards (it used to be evicted).
Effects
POST .../effectwipe_left/right/up/down,iris_open/close,barn_doors,luma_wipe,clock_wipe,blinds,checker_wipe,noise_dissolve,melt,heart_iris,star_wipe,pinwheel,crosshatch,hex_dissolve,warp_wipeglitch_cut,flash_dissolve,whip_pan_left/right,punch_zoom,pixelate_take,zoom_blur,spin,tv_roll,negative_flash,ripple— full-frame envelope over a delayed cut / fade / pushAll shader transitions downgrade to Fade on the CPU backend or with Shader FX off; PiP-aware takes keep master envelopes on top of the fade.
Animation model
Shaders are self-animating:
glshaderprovides atimeuniform (buffer PTS) per frame; Rust programsu_start/u_durationonce per take and the shader computes its own progress — frame-accurate at composite time, zero per-frame CPU, no GstController involvement.Hard-won correctness details (each found against real deployments)
create-shadersignal — glshader'sfragmentproperty is inert once a shader exists. The attached handler compiles on the GL thread and keeps the previous shader on compile failure (a runtime swap can never kill the pipeline).u_invertparticipates in the uniform-only neutral reset — uniform values persist on the GL program and a leftover inverted mask would black out the branch.Operator page
The pickers are tiered: production staples (directional/iris/barn/luma wipes, flash take, chroma key / blur / pixelate / vignette looks) are visible by default; novelty shapes and the remaining master FX sit behind a MORE toggle (persisted per browser), stylization looks under a "Stylized" optgroup. The full catalog stays available — and CI-validated — but the default view reads like a production switcher.
LOOKS panel with live parameter controls (param-only changes swap uniforms without recompiling), FX UI hidden when the engine is unavailable, state sync via
/state+ newVisionMixerEffectChangedWS event.Testing
shader_validation_test: every composed fragment compiles and runs in a real GL pipeline (llvmpipe-compatible, runs in CI); pixel-checked regression test that runtime fragment swaps actually take effectvision_mixer_fx_test: end-to-end GPU pipeline — FX slots, looks (clamping, param-only updates, invalid input rejection), wipe + master takes, master look asserted to survive a master-FX take, pipeline stays PLAYING; letterbox repro test pixel-verifies both wipe orientations animate and land on the right sourcepipeline_lifecycle_testgreen (the FX slots add elements with signal handlers — no leak regressions)🤖 Generated with Claude Code