Veda: Scalable Video Diffusion via Distilled Sparse Attention

Waver-T2V-12B · 720P · 241 frames: FlashAttention-3 vs. Veda

FlashAttention-3 · full attention 0.0min

Done · now playing

Veda · Ours 95% sparsity 0.0min

Done · now playing

5.1× end-to-end acceleration 19.4 min → 3.8 min 95% sparsity · quality preserved

Scaling Law in Practice

Tile skipping changes the long-video scaling curve

Full attention: O(N²) over spatiotemporal tokens.
Veda: hardware-aligned Top-k tile skipping.
245K tokens: 1576 ms → 309 ms per layer.

5.1×

End-to-end acceleration
Waver-T2V-12B · 720P · 241f

10.5×

Self-attention acceleration
against the FlashAttention-3 dense baseline

ScalingPer-layer latency as sequence length grows. Full attention follows the O(N²) reference; Veda tracks a near-linear trend and reaches a 5.1× speedup at 245K tokens.

Attention Structure Distillation

Recover the distributional geometry of full attention

Teacher: Max-pooled full attention converts dense attention into tile-level supervision, preserving high-response dependencies that averaging would dilute.
Estimator: TripPool + head-specific Q/K projections reconstruct the dense tile distribution from average, maximum, and minimum statistics.
Execution: Head-aware Top-k selection produces a tile mask, which is passed to a tile-skipping kernel to fetch only selected K/V tiles.

90%

Sparsity that matches full attention
WaverBench1.0 human side-by-side

92→50%

Attention bottleneck reduced
per-layer runtime share

ArchitectureVeda compared with static sparse patterns (SVG) and implicitly learned dynamic masks (VSA). Solid lines denote inference data flow; dashed lines denote training-only supervision. Hover a component card below, or any block in the diagram, to highlight the corresponding stages.

Principle 01

Full-Attention Structure Distillation

Max-pooled full attention provides a tile-level teacher distribution. KL distillation aligns Veda with the dense attention geometry before the binary sparse mask is materialized.

Principle 02

Head-Conditioned Tile Geometry

Per-layer, per-head tile shapes (p_t, p_h, p_w) adapt the tile geometry to heterogeneous spatial and temporal attention heads under a fixed hardware budget.

Principle 03

Tile-Sparse Kernel

A ThunderKittens kernel fetches only selected K/V tiles via TMA and overlaps sparse gathers with WGMMA compute, turning structural sparsity into wall-clock acceleration.

Evaluation

Perceptual fidelity under practical acceleration

Human side-by-side evaluation on WaverBench1.0, a 304-prompt benchmark centered on object motion, camera motion, and temporal consistency, together with latency under identical sparse-kernel infrastructure. Use the human-study tabs to switch comparison targets.

Human Evaluation · WaverBench1.0

90% sparse attention remains preference-matched to full attention

Against Full Attention: visual quality is essentially tied (50% vs. 49%), with 63% ties on motion.
Against VSA@95%: Veda leads 76% vs. 24% in visual preference and 39% vs. 16% overall.

User studyBlinded side-by-side win / tie / loss across four axes on WaverBench1.0 (Waver-T2V-1B, 480P, 81 frames). Tabs compare Veda with dense attention and VSA under different sparsity budgets.

ms / layerDense FlashAttention-3 vs. Veda, including MLP, sparse-mask preparation, and the attention kernel.

Inference Latency

Wall-clock speedups include mask prediction

Per-layer latency is reported against FlashAttention-3 at 720P and 480P across Waver and Wan2.1. The accounting includes MLP, mask preparation, and sparse-kernel execution, so the gain is measured at the layer level rather than inferred from FLOPs.

Waver-T2V 720P: 315 → 78 ms per layer · 4.03×.
Wan2.1 720P: 584 → 221 ms per layer · 2.64×.
Attention share of compute: 92% → 50%.

Qualitative Comparison

Matched generations under dense and sparse attention

Prompt- and seed-paired generations isolate the effect of the attention operator. Drag the handle to compare geometry, motion, and prompt alignment; select clips from the thumbnails below.

Video samples under Veda's sparse attention

All clips are generated with Veda. We use 90% sparsity for Waver-T2V-1B at 480P / 81 frames, and 95% sparsity for Waver-T2V-12B at 720P / 241 frames. Across portraits, animals, landscapes, sports, robotics, and cinematic camera motion, these samples show that Veda preserves scene structure and temporal coherence across diverse generation regimes. Click any clip to enlarge.

Empirical Study

What matters for a good sparse attention model?

The central question is not how many tiles can be removed, but whether the retained mask preserves the tile-wise attention structure induced by the dense softmax distribution.

01 · Empirical Failure Mode

Structural artifacts indicate mask-induced connectivity errors

When sparsity exceeds the moderate regime, prior sparse-attention methods do not simply lose texture or prompt fidelity. They produce a distinct failure mode: water-ripple warping across surfaces, local geometric distortion, and temporal flicker between adjacent frames. These artifacts indicate broken spatiotemporal connectivity in the attention graph, rather than ordinary semantic hallucination from the dense generator.

Pooling-based masks produce water-ripple distortions and temporal flicker, revealing structural failures in sparse token connectivity.

02 · Controlled Mask Intervention

Tile-wise attention structure, rather than sparsity alone, determines fidelity

To separate sparsity from mask quality, we hold the budget fixed at 90% sparsity and change only the selected tile mask. The Oracle mask, obtained by max-pooling the full-attention map and retaining the high-response key tiles for each query tile, preserves high-fidelity structure. Average pooling, under the same budget, fails. This controlled comparison shows that degradation is governed by structural alignment with full attention. Tile recall makes the gap measurable: Veda recovers 66.4% of oracle-selected tiles, whereas average pooling recovers 34.2%.

Full Attention

Oracle Mask

Average Pooling

Veda · Ours

Same 90% sparsity budget. Full Attention, Oracle, and Veda preserve structure; Average Pooling drifts away from the full-attention geometry. Tile recall: 66.4% for Veda vs. 34.2% for average pooling.

Attention maps across heads and timesteps showing diverse spatial and temporal footprints.

03 · Head-Dependent Attention Geometry

Uniform tiling misaligns heterogeneous heads

Video DiT heads are not exchangeable. Some heads concentrate on local spatial structure, while others connect temporally distant regions; these patterns also evolve across diffusion timesteps. A uniform tile layout can split high-affinity neighborhoods for one head and merge unrelated tokens for another, increasing intra-tile variance and reducing oracle-mask recall.

Head-Aware Tile Search

Optimized tile factorizations under a fixed hardware budget

Candidate factorizations satisfy p_tp_hp_w = B, preserving the hardware tile size.
For each layer and head, Veda selects the tiling that minimizes ||O^fu − O^sp(π)||_F² on calibration samples.
Net human preference over the best static tiling [4,4,8]: +9.5% Overall, +7.2% Motion.

AblationNet human-preference win rate against the strongest static tiling baseline, showing that head-aware geometry is not cosmetic; it improves motion and overall quality.

Theoretical Rationale

Sufficient conditions for stable tile sparsification

The derivation follows the view of attention as a probability distribution over keys: the softmax row is not just a heuristic normalization, but an entropy-regularized geometry that must be preserved before sparsification.

1

Softmax Geometry

Missing high-energy keys induces distributional shift

For one query, attention is the gradient of a Log-Sum-Exp potential over logits u_j = q·k_j. Equivalently, each row is an entropy-regularized transport distribution over keys: it balances similarity to the query with entropy. Dropping a dominant key therefore changes the normalization constant rather than adding a small local perturbation. The remaining keys are renormalized by Z/Ẑ > 1, so probability mass is reassigned to lower-energy or irrelevant tokens. At video scale, this produces coherent structural errors: the sparse model follows the wrong connectivity graph with high confidence.

Ψ(u)=τ·log Σ_j exp(u_j/τ) ⟹ P̃_j = (Z / Ẑ) · P_j > P_j

↓ therefore, sparse attention first needs to preserve the dominant tile connectivity of full attention

2

Tile Statistics

Mean pooling discards second-order score information

A tile score must be estimated from compressed query and key representations. Average pooling preserves only the centroid. However, the dot product decomposes into a mean-product term and a covariance term; two tiles with similar means can still have different high-frequency structure and therefore different attention scores. Veda augments the mean with extrema. By the Bhatia-Davis inequality, the variance of a bounded variable is controlled by its mean, maximum, and minimum, so {avg, max, min} provides a compact estimate of both centroid and spread.

q·k = d·μ_qμ_k + d·Cov(q,k) · σ² ≤ (M−μ)(μ−m)

↓ this estimator is reliable only when tokens within each tile have low variance

3

Tile Geometry

Low-variance tiling tightens the approximation bound

Let each token be decomposed into a tile mean plus a residual. The score error induced by compressing a tile is bounded by the product of query- and key-side intra-tile standard deviations. Under 3D RoPE, video tokens lie on a spatiotemporal manifold, but different heads induce different anisotropic metrics on that manifold. A fixed raster layout can scatter high-affinity temporal neighbors or merge unrelated spatial regions. Head-Aware Tiling searches (p_t, p_h, p_w) per head, minimizing the sparse-output error and reducing the variance term that controls score estimation.

E = |Σ_i ⟨r^q_i, r^k_i⟩| ≤ B · σ_q · σ_k

↓ with compact statistics and low-variance tiles, the remaining question is how to train the dense attention geometry before discrete Top-k

4

Distillation Objective

KL alignment learns the pre-Top-k attention geometry

Top-k selection is discrete, but the attention distribution that induces it is continuous. Veda therefore supervises the estimator before thresholding: max-pooled full attention defines the target tile distribution, and the estimator produces a soft distribution over key tiles. Minimizing row-wise KL divergence aligns the information geometry of the predicted distribution with the full-attention teacher. Locally, KL is a Fisher-weighted quadratic form, so errors are penalized in the natural geometry of the simplex rather than in raw Euclidean score space. After this continuous alignment, Top-k selection can be applied as a hardware-executable binary mask.

L_distill = D_KL(A_tgt || A_pred) ≈ 1/2 · Δθ^TI(θ)Δθ

Takeaway

Stable high-sparsity attention requires four aligned choices: preserve the oracle attention structure, estimate tile scores with sufficient statistics, group tokens with head-aware low-variance tilings, and distill the dense probability geometry before Top-k discretization. Veda combines these conditions to match full attention at 90% sparsity on WaverBench1.0 and scale to 95% sparsity for 720P / 241-frame generation.

Citation

BibTeX

@inproceedings{han2026veda,
  title     = {Veda: Scalable Video Diffusion via Distilled Sparse Attention},
  author    = {Han, Shihao and Yang, Hao and Mei, Xiaofeng and Hu, Xinting and Jiang, Yi and Qi, Xiaojuan},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2026}
}