Commit 29c7a33
committed
Refactor attention subsystem into 4-layer Onion Architecture (Phases 1-6)
- Extract legacy UNet transformer blocks from attention_flax.py into src/maxdiffusion/models/unet_transformer_blocks_flax.py (-68% LOC reduction in attention_flax.py).
- Create src/maxdiffusion/models/attention_strategies/ package with communication-only strategies (SingleShardStrategy, RingAttentionStrategy, UlyssesStrategy, CustomRingAttentionStrategy) and Protocol definition.
- Introduce extensible KERNEL_REGISTRY in attention_utils.py and attention_dispatch.py with @register_kernel decorators for all 13 attention backends.
- Implement robust memory-efficient attention chunking and ValueError tensor validation in attention_dispatch.py.
- Implement scan-based ring communication loops (length=ring_size-1) without static loop unrolling in custom_ring.py to keep compile times and HLO graph size minimal.
- Ensure exact cross-attention sequence length and value head dimension handling across Ulysses and Ring strategies.
- Maintain 100% mathematical and operational parity with origin/main across all attention kernels (dot_product, Pallas splash, TokaMax ring, Ulysses, and custom dense splash kernels).
- Add unit test suite src/maxdiffusion/tests/attention_strategies_test.py covering strategy instantiation, hooks, block size adapters, and kernel registry validation.1 parent df69836 commit 29c7a33
17 files changed
Lines changed: 3967 additions & 3054 deletions
File tree
- src/maxdiffusion
- kernels
- splash_attention
- models
- attention_strategies
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
66 | | - | |
| 66 | + | |
| 67 | + | |
67 | 68 | | |
68 | 69 | | |
69 | 70 | | |
| |||
815 | 816 | | |
816 | 817 | | |
817 | 818 | | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
Lines changed: 7 additions & 463 deletions
Large diffs are not rendered by default.
0 commit comments