AI Insight
This study introduces Mixture-of-Depths Ensemble (MoDE), a new approach for fine-tuning large language models that leverages predictions from multiple late layers rather than relying solely on the final layer. The researchers found that intermediate layers in transformer models already contain well-formed, task-relevant predictions, and by treating these late layers as an ensemble with learned routing weights, they achieved comparable or slightly improved performance with minimal additional parameters. MoDE can be combined with existing tuning methods like LoRA to modestly enhance reasoning capabilities.
Why it matters
This approach could make fine-tuning large language models more efficient by utilizing information already present in intermediate layers, potentially reducing computational costs while maintaining or improving model performance. The technique offers a practical way to enhance existing tuning methods without requiring substantial architectural changes or parameter increases.
Understand the Science
-cross
Abstract: Transformer-based Large Language Models (LLMs) traditionally rely on final-layer loss for finetuning and final-layer representations for predictions, potentially overlooking the predictive power embedded in late layers. Interpretability tools such as the logit lens show that late-layer representations already carry largely formed, task-relevant predictions; here we ask whether that observation can be turned into an actionable training signal. We find that focusing tuning effort on these layers can yield losses comparable to those of the final layer, with complementary test-time behaviour. Building on this, we introduce a tuning framework, Mixture-of-Depths Ensemble (MoDE), which treats the late layers as an ensemble that contributes to the final logits through learned routing weights. MoDE can be applied on top of any existing tuning method (e.g., LoRA) and, in our experiments, modestly improves reasoning performance at a small parameter overhead. We present MoDE as a mechanism study showing that late-layer logits can be made directly useful for tuning, and that they can substitute for substantially larger trainable modules with comparable performance.
Source: Tuning Language Models by Mixture-of-Depths Ensemble