A Proof of Learning Rate Transfer Under $μ$P

Soufiane Hayou
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:3322-3330, 2026.

Abstract

We provide the first proof of learning rate transfer with width in a linear multi-layer perceptron (MLP) parametrized with $\mu$P, a neural network parameterization designed to “maximize” feature learning in the infinite-width limit. We show that under $\mu$P, the optimal learning rate converges to a \emph{non-zero constant} as width goes to infinity, providing a theoretical explanation to learning rate transfer. In contrast, we show that this property fails to hold under alternative parametrizations such as Standard Parameterization (SP) and Neural Tangent Parametrization (NTP). We provide intuitive proofs and support the theoretical findings with extensive empirical results.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-hayou26a, title = { A Proof of Learning Rate Transfer Under $μ$P }, author = {Hayou, Soufiane}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {3322--3330}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/hayou26a/hayou26a.pdf}, url = {https://proceedings.mlr.press/v300/hayou26a.html}, abstract = { We provide the first proof of learning rate transfer with width in a linear multi-layer perceptron (MLP) parametrized with $\mu$P, a neural network parameterization designed to “maximize” feature learning in the infinite-width limit. We show that under $\mu$P, the optimal learning rate converges to a \emph{non-zero constant} as width goes to infinity, providing a theoretical explanation to learning rate transfer. In contrast, we show that this property fails to hold under alternative parametrizations such as Standard Parameterization (SP) and Neural Tangent Parametrization (NTP). We provide intuitive proofs and support the theoretical findings with extensive empirical results. } }
Endnote
%0 Conference Paper %T A Proof of Learning Rate Transfer Under $μ$P %A Soufiane Hayou %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-hayou26a %I PMLR %P 3322--3330 %U https://proceedings.mlr.press/v300/hayou26a.html %V 300 %X We provide the first proof of learning rate transfer with width in a linear multi-layer perceptron (MLP) parametrized with $\mu$P, a neural network parameterization designed to “maximize” feature learning in the infinite-width limit. We show that under $\mu$P, the optimal learning rate converges to a \emph{non-zero constant} as width goes to infinity, providing a theoretical explanation to learning rate transfer. In contrast, we show that this property fails to hold under alternative parametrizations such as Standard Parameterization (SP) and Neural Tangent Parametrization (NTP). We provide intuitive proofs and support the theoretical findings with extensive empirical results.
APA
Hayou, S.. (2026). A Proof of Learning Rate Transfer Under $μ$P . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:3322-3330 Available from https://proceedings.mlr.press/v300/hayou26a.html.

Related Material