AI Insight
This paper examines scaling properties of the muon optimizer, specifically focusing on its iterative orthogonalization procedure. The authors demonstrate both theoretically and empirically that singular values of random matrices decrease as scale increases, which may pose challenges for muon's orthogonalization step at larger model sizes. The work documents this scaling behavior but does not propose solutions to address the identified issue.
Why it matters
As the muon optimizer gains traction as a potential alternative to Adam in large-scale machine learning, understanding how its components scale is critical for practical deployment. This finding suggests potential limitations that practitioners should be aware of when training very large models with muon.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: The muon optimizer has picked up much attention as of late as a possible replacement to the seemingly omnipresent Adam optimizer. Recently, care has been taken to document the scaling laws of hyper-parameters under muon such as weight decay and learning rate. However, at much larger scales the iterative orthogonalization procedure present in muon may suffer a possible issue as the singular values of random matrices shrink with scale. This paper shows this scaling behavior theoretically and empirically on random matrices but does not suggest what to do about it.