AI Insight
This study investigates when the Muon optimizer outperforms AdamW in reinforcement learning fine-tuning of language models. Testing on ALFWorld tasks with Qwen2.5 models (0.5B-3B parameters), researchers found that Muon enables more aggressive stable learning steps under matched KL divergence and clipping constraints, improving late-stage task success rates at higher learning rates (3×10^-5) compared to standard AdamW (10^-6). However, this advantage diminishes when AdamW is properly tuned, when models approach performance saturation, or when update magnitudes are matched, indicating the benefit stems from Muon's scaling convention rather than fundamental optimizer superiority.
Why it matters
The findings provide practical guidance for selecting optimizers during post-training of AI agents, potentially reducing computational costs by enabling faster convergence in specific operating regimes. This work helps researchers understand when aggressive optimization steps are beneficial versus when careful tuning of standard methods suffices.
Understand the Science
arXiv:2607.16169v3 Announce Type: replace
Abstract: Muon is competitive with AdamW in large-scale pre-training, but its operating regime in reinforcement-learning post-training remains unclear. We map this regime on ALFWorld, a sparse-reward agentic benchmark, using three group-based objectives and Qwen2.5 models from 0.5B to 3B. Under a shared KL and clipping recipe, matched optimizer comparisons and AdamW learning-rate controls trace the usable step-size range. AdamW responds non-monotonically to its learning rate, whereas fan-in Muon remains stable at a more aggressive effective step. At a learning rate of 3 x 10^-5, it improves late success over an AdamW 10^-6 baseline after correction across rate-metric tests. Its normalized-AUC effect is directionally positive but less uniform; the heuristic-matched lower-rate effect is less consistent, and tuned AdamW nearly matches high-rate Muon on 3B GraphGPO. High-rate Muon applies 3.53 times AdamW’s hidden-matrix update RMS, while a full-budget RMS-matched control removes the late-success gain. Together, these results identify a recipe-level operating regime in which fan-in Muon supports a more aggressive stable effective step under shared KL and clipping. The margin is largest when optimization headroom remains and contracts near saturation, after AdamW tuning, or under magnitude matching. The scale-matched control ties this spectral effect to Muon’s scaling convention rather than establishing a universal optimizer ranking.