DIET: DeletIon-response Expert Trimming for Video Diffusion Transformers

Training-Free Expert Pruning

Expert pruning shrinks an already-trained mixture-of-experts model without retraining, distillation or gradients—but deleting an expert re-routes the tokens it served, so scoring experts in isolation measures a quantity the model never uses. We make the counterfactual cheap to observe: one all-expert calibration capture turns every layer-local leave-one-out deletion into a cache-based replay, and a single deletion-response signature per expert drives the selection. We prune LingBot-Video 30B-A3B at a 50% budget with no training: a pruned replica fits a single 48 GB card while generation quality remains essentially intact under the official scorer.

Expert pruning is a re-routing problem, not a subtraction problem. (a) Deleting an expert changes the grouped top-k decision: the tokens it served are re-routed. (b) One instrumented all-expert capture makes every layer-local deletion replayable at the captured states, so the whole 6,144-deletion study costs no further forward passes. (c) At a 50% budget this yields a smaller checkpoint and a rise in the official VBench Total, without any training.

Abstract

Video diffusion transformers (DiTs) increasingly rely on mixture-of-experts (MoE) architectures, where only a sparse subset of a large expert bank is activated per token. While dynamic sparsity reduces active compute, it leaves the full expert storage footprint intact. Furthermore, conventional one-shot pruning criteria rely on static activation or routing statistics, failing to capture the layer-level re-routing behavior triggered after expert deletion. To make this post-deletion behavior observable without prohibitive cost, we record all expert outputs and router states across matched conditional and unconditional tokens in a single all-expert calibration pass. Consequently, replaying any single-expert deletion and its grouped re-routing reduces to tensor arithmetic on cached states, requiring zero additional model forward passes. Using these deletion responses, we formulate retained-expert selection as an Overall Diversity Loss (ODL): each pruned expert is matched with its nearest retained expert and the summed nearest-neighbor cosine distances define a set-level coverage loss under which deleted experts closer to retained ones cost less, directly prioritizing functional coverage. We solve this optimization in two stages: an intra-layer local search combining greedy initialization, single-swap refinement, and simulated annealing to select retained set under a fixed retention ratio, and an inter-layer regression-guided search that optimizes expert-count allocation across layers. Evaluated on LingBot-Video 30B-A3B, pruning 50% of the experts (6,144 → 3,072) without fine-tuning reduces the checkpoint footprint from 57 GB to 30 GB, enabling single-card deployment on a 48 GB GPU while generation quality remains essentially intact: we observe the official VBench Total rise from 0.7941 to 0.8115 across a fixed 284-case protocol. Across all other tested retention budgets, DIET consistently outperforms competitive baselines adapted from large language models.

Method

One instrumented capture. A single pass over 120 calibration cases—evaluating all 128 experts per layer on matched conditional and unconditional tokens—records every expert output and every router score (368,640 matched token pairs), once.
Frozen-state replay. Deleting an expert and recomputing the grouped top-k routing at the captured states reproduces the layer-local leave-one-out deletion with zero additional forward passes: all 6,144 deletions become cache arithmetic.
Deletion-response signatures. Each expert is summarised by the change it causes when removed. Signatures are used strictly for their directional orientation: concurrent removals violate linear superposition, so absolute perturbation size does not reliably reflect joint dynamics, while direction measures functional interchangeability.
Overall diversity loss selection. The retained set is chosen to minimise an overall diversity loss (ODL)—each deleted expert pays the distance to its nearest retained expert in deletion-response space—under the grouped-router feasibility constraints, preserving diverse deletion responses rather than merely active experts.
Regression-guided budget. Per-layer kept counts come from an offline search on a 142-case budget-search set that linearises the official VBench Total; the 284-case protocol's 142 strictly held-out prompts are evaluated separately.

Results

50%

of the expert bank removed (6,144 → 3,072 experts) with no retraining

1× 48 GB

a pruned replica fits one card; replica VRAM halves (93 → 45 GiB)

0.8115

VBench Total (pruned) vs 0.7941 (unpruned) under the official scorer

Main results. (a) The selection-corpus sweep: the best operating point occurs at 50% retention. (b) The full 284-case protocol at the same budget, all methods at the identical (searched) per-layer kept-count vector. (c) Paired per-case differences against the unpruned model and the transplanted criteria, with 95% bootstrap intervals.

Analysis

Breakdown of additive superposition under joint multi-expert deletions

What breaks under joint deletions is additivity, not distance. Joint multi-expert removals violate linear superposition: the cosine similarity between a joint deletion perturbation and the sum of its single-deletion responses falls from 1.00 at k = 1 to 0.66 at k = 64, with reconstruction error rising to 0.95. Objectives built on summed responses degrade by 5.44 points, consistent with the non-additivity measured here.

Pruning is not routing preservation. The best-scoring mask is the most disruptive to the original routing, while the criteria that lose the most quality preserve routing the most—so minimising routing change is the wrong objective.

Qualitative Results

One case per VBench dimension (the instances follow the paper's qualitative grids), generated from identical prompts and seeds: the pruned model at 50% retention against the unpruned model. The final three blocks extend the comparison to all arms on three representative cases, where the ported criteria degrade structurally while DIET remains visually consistent with the unpruned model.

Subject consistency · “an elephant taking a peaceful walk”