AI Insight
This paper addresses the problem of reducing memory requirements in Mixture-of-Experts (MoE) language models by systematically analyzing how to select which experts to prune. The researchers develop a framework based on three factors—routing frequency, gate weighting, and activation strength—and propose two new pruning criteria called Mean Activation Norm (MAN) and Mean Squared Activation Norm (MSAN). Testing across four MoE models and 16 benchmarks shows these new criteria improve performance by an average of 4.0 points over existing methods when pruning experts without task-specific information.
Why it matters
This research provides practical guidance for deploying large MoE language models with reduced memory footprint, making them more accessible for real-world applications. The systematic framework helps practitioners choose appropriate pruning strategies based on whether they need general-purpose or task-specific model compression.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Mixture-of-Experts (MoE) language models reduce per-token computation through sparse expert activation, yet deployment still requires storing the full expert pool, making one-shot expert pruning a practical approach for reducing memory usage. Although effective, existing criteria are largely heuristic, and no single criterion is universally optimal. Thus, establishing a principle for selecting pruning criteria suited to different deployment objectives remains an important yet largely underexplored problem in one-shot expert pruning. To this end, we introduce a systematic formulation for one-shot MoE expert pruning organized around three factors: routing frequency, gate weighting, and activation strength. The formulation yields a criteria selection principle: task-agnostic pruning should favor routed-token-averaged, gate-free activation-based criteria, whereas task-specific pruning can benefit from retaining routing-frequency and gate-weight information. Beyond this principle, the formulation also provides a systematic view of existing heuristic criteria and gives rise to two new task-agnostic criteria, Mean Activation Norm (MAN) and Mean Squared Activation Norm (MSAN). Across four representative MoE models and 16 diverse benchmarks, MAN and MSAN are consistently strong in the task-agnostic setting, obtain the top-two average ranks, and improve average performance by 4.0 points on average and up to 8.8 points over the strongest baseline.