AI Insight
This study demonstrates that while layer pruning can effectively compress large language models for classification tasks with minimal performance loss, it significantly impairs performance on generative reasoning tasks like mathematical problem-solving and code generation. The researchers found that even with supervised finetuning on task-specific data, pruned models recover up to 90% performance on classification but show persistent deficits in algorithmic capabilities such as arithmetic computation and structured text generation. Analysis of models trained with nearly 100 billion post-pruning tokens confirms these limitations persist even on simple arithmetic tasks.
Why it matters
These findings reveal critical constraints for model compression techniques in resource-limited settings, showing that layer pruning may not be suitable for applications requiring mathematical reasoning or code generation. This has important implications for deploying compressed AI models in educational tools, programming assistants, and other applications requiring multi-step logical reasoning.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Recent work has shown that layer pruning can effectively compress large language models (LLMs) while retaining strong performance on classification benchmarks, often with little or no finetuning. In contrast, generative reasoning tasks, such as GSM8K and HumanEvaltextsuperscript{+}, exhibit substantially weaker recovery. We show that beyond surface-level text degradation, pruning leads to a loss of key algorithmic capabilities, including arithmetic computation and balanced parenthesis generation. Under realistic post-training constraints, using a single 80GB GPU and without access to pretraining-scale data or compute, we evaluate a simple recovery strategy based on supervised finetuning with self-generated responses. This approach recovers up to 90% of baseline performance on classification tasks, but recovery for generative reasoning remains limited. We further find that this gap persists even under a favorable task-aligned recovery setting, where pruned models are fully finetuned on self-generated GSM8K responses, suggesting that the degradation is not merely due to generic instruction data or parameter-efficient tuning. As complementary evidence, we analyze a depth-pruned model trained with nearly 100B post-pruning tokens and find that deficits persist even on simple arithmetic tasks that do not require multi-step generation. Overall, we characterize practical recovery limits of layer pruning for generative reasoning and provide guidance on when depth reduction is effective under constrained post-training regimes.
Source: On the Limits of Layer Pruning for Generative Reasoning in Large Language Models