Commit 7d1b25d
committed
Refactor attention subsystem into 4-layer Onion Architecture (Phases 1-6)
- Extract legacy UNet transformer blocks from attention_flax.py into src/maxdiffusion/models/unet_transformer_blocks_flax.py (-68% LOC reduction in attention_flax.py).
- Create src/maxdiffusion/models/attention_strategies/ package with communication-only strategies (SingleShardStrategy, RingAttentionStrategy, UlyssesStrategy, CustomRingAttentionStrategy) and Protocol definition.
- Introduce extensible KERNEL_REGISTRY in attention_utils.py and attention_dispatch.py with @register_kernel decorators for all 13 attention backends.
- Implement robust memory-efficient attention chunking and ValueError tensor validation in attention_dispatch.py.
- Implement scan-based ring communication loops (length=ring_size-1) without static loop unrolling in custom_ring.py to keep compile times and HLO graph size minimal.
- Ensure exact cross-attention sequence length and value head dimension handling across Ulysses and Ring strategies.
- Maintain 100% mathematical and operational parity with origin/main across all attention kernels (dot_product, Pallas splash, TokaMax ring, Ulysses, and custom dense splash kernels).
- Add unit test suite src/maxdiffusion/tests/attention_strategies_test.py covering strategy instantiation, hooks, block size adapters, and kernel registry validation.1 parent df69836 commit 7d1b25d
18 files changed
Lines changed: 3989 additions & 3055 deletions
File tree
- src/maxdiffusion
- kernels
- splash_attention
- models
- attention_strategies
- tests
- ltx2
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
49 | 65 | | |
50 | 66 | | |
51 | 67 | | |
| |||
63 | 79 | | |
64 | 80 | | |
65 | 81 | | |
66 | | - | |
| 82 | + | |
| 83 | + | |
67 | 84 | | |
68 | 85 | | |
69 | 86 | | |
| |||
815 | 832 | | |
816 | 833 | | |
817 | 834 | | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
Lines changed: 7 additions & 463 deletions
Large diffs are not rendered by default.
0 commit comments