Track4D-GS

Canonical-anchor track supervision for dynamic Gaussian reconstruction

Track4D-GS is a reproducible research prototype that adds long-term 2D motion supervision to SC-GS without replacing its native sparse control-node warp. Each frozen association stores a canonical Gaussian anchor together with the Gaussian's original node indices and blend weights. The anchor is deformed through the native SC-GS warp and only then projected into the camera.

Scope: Gate 3 passed on one controlled procedural scene. This repository does not claim real-world generalization, arbitrary-video reconstruction, multi-object decomposition, or state-of-the-art performance.

Watch: 40-second technical demo · 20-second social cut

Result

Against an equal-budget Stabilized SC-GS branch initialized from the same checkpoint, Track4D-GS improved every reported held-out motion metric:

Metric	Stabilized SC-GS	Track4D-GS	Change
Held-out mean 2D error	25.919 px	21.080 px	-18.7%
Held-out endpoint error	58.983 px	45.585 px	-22.7%
First-reappearance error	15.049 px	10.556 px	-29.9%
Residual velocity jitter	1.109	0.906	-18.3%
Test PSNR	15.939 dB	15.877 dB	-0.062 dB
Test SSIM	0.7007	0.7084	+0.0077

The matched branches each ran 1,000 successful updates. Peak reserved VRAM was 4.324 GiB for the baseline and 4.193 GiB for Track4D-GS. The complete machine-readable aggregate is in artifacts/final_results.json; published media hashes and frame counts are in artifacts/demo_manifest.json.

Method

reference-frame 2D observation
  -> frozen nearest valid projected dynamic Gaussian
  -> canonical position + native node IDs + native blend weights
  -> native SC-GS ControlNodeWarp at time t
  -> audited camera projection
  -> visibility-masked robust 2D loss

Direct weighted node projection is retained only as a baseline/ablation. Training and held-out track IDs are disjoint, and the main motion claims are computed on tracks that receive no optimization supervision. Occlusion recovery is measured on the first reappearance frame and the first three visible frames after reappearance.

What was engineered

• Canonical anchors that preserve native SC-GS sparse control-node semantics.
• A synthetic Gate 2.5 fixture covering camera projection, coordinate transforms, visibility masking, fixed association behavior, and gradients.
• A disabled path that skips track loading, association, deformation, and projection when lambda_track=0.
• Exact-resume staged training with per-loss logging and memory/geometry guards.
• A quaternion-valid smooth activation bridge for a diagnosed SC-GS hard activation failure.
• Versioned, pickle-free NPZ track tensors with separate JSON manifests.
• Disjoint train/held-out motion and occlusion-recovery evaluation.

Controlled benchmark

• 96 chronological 400x400 training frames
• 24 validation and 24 test views
• exact RGB, alpha, dynamic mask, metric depth, part IDs, and cameras
• 20,000 initialization points and 128 SC-GS control nodes
• 96 training tracks and 32 disjoint held-out tracks
• 79 total occlusion-recovery events
• one dominant four-part articulated foreground object
• no downloaded meshes, textures, music, or visual assets

The procedural scene is deliberately high-frequency and visually diagnostic: it was designed to expose camera, association, and deformation errors rather than serve as a photorealistic benchmark.

Repository layout

artifacts/             machine-readable accepted results
assets/                compact portfolio demo and result thumbnail
configs/               baseline and method configurations
environment/           resolved WSL2/CUDA environment snapshots
patches/sc_gs/         minimal patches against the pinned SC-GS submodule
reports/               specifications, audits, failures, and final results
scripts/               data, training, evaluation, and demo entry points
src/track4dgs/         first-party Python package
tests/                 CPU and CUDA-aware regression tests
third_party/           pinned Git submodules

Checkout and lightweight validation

Clone recursively:

git clone --recursive https://github.com/haoranli623/Track4D-GS.git
cd Track4D-GS
git -C third_party/sc_gs apply \
  ../../patches/sc_gs/0001-fix-dnerf-rgba-uint8.patch
git -C third_party/sc_gs apply \
  ../../patches/sc_gs/0002-track4dgs-headless-staged-hooks.patch

The recorded stack uses WSL2 Ubuntu 22.04, Python 3.10, PyTorch 2.1.2+cu121, and CUDA 12.1. See environment/README.md before building the pinned CUDA extensions.

With the project environment active, run the first-party regression suite:

PYTHONPATH=src:scripts/gate1 \
python -m unittest discover -s tests -v

The publication baseline passes 80 tests. GPU-specific tests automatically skip when CUDA is unavailable.

Full training requires regenerating the procedural dataset and producing the staged SC-GS checkpoints. Exact commands and accepted artifact identities are recorded in reports/EXPERIMENT_REPRODUCTION.md. Large datasets, checkpoints, raw run directories, and the complete 250 MB editorial package are intentionally excluded from Git.

Read next

• Final project results
• Experiment reproduction
• Activation failure audit
• Gate 2.5 analytic fixture
• Project specification
• Resume and demo summary

Limitations

• Evidence is limited to one controlled procedural scene.
• Reference-image nearest-neighbor association is subpixel accurate but often selects the wrong canonical surface identity; exact held-out identity was 37.5% in the post-freeze GT audit.
• Exact upstream SC-GS hard activation failed on this scene. Stabilized SC-GS is a distinct baseline, not an unmodified upstream success.
• Strict disabled-path parameter/loss equivalence passed, while a requested cross-run per-pixel render threshold retained sparse CUDA raster-boundary differences.
• No real-world dataset, multi-scene result, LPIPS result, or optional ablation is claimed.

License

First-party Track4D-GS code and documentation are released under the MIT License. Third-party repositories retain their own licenses; notably, the optional CoTracker submodule is CC BY-NC 4.0. See THIRD_PARTY_NOTICES.md.