Understanding MARS: When Scaling Momentum Correction Provably Helps

Egor Shulgin, Tamaz Gadaev, Sarit Khirirat, Peter Richtárik
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:112936-112965, 2026.

Abstract

MARS (Yuan et al., 2025) has recently emerged as a strong optimizer for large language model (LLM) training by scaling the correction term in momentum-based variance reduction (MVR). However, existing theory does not explain why this modification can improve convergence over the unscaled MVR choice $\gamma=1$. In this paper, we provide a theoretical explanation for this phenomenon. We introduce $\gamma$-similarity, a refined similarity condition that captures how the scaling coefficient interacts with the stochastic gradient-difference structure. This condition recovers standard similarity at $\gamma=1$ and smoothness at $\gamma=0$. Using $\gamma$-similarity, we derive convergence guarantees for fixed-$\gamma$ MARS whose complexity depends explicitly on $\gamma$ and the corresponding $\gamma$-similarity constant. The bound reveals why small values of $\gamma$ can be beneficial: they may reduce the similarity term enough to outweigh the penalty from deviating from MVR. We prove that optimizing $\gamma$ gives MARS a lower complexity guarantee than MVR. Experiments with MARS-AdamW on GPT-style LLM pretraining corroborate the theory, showing that properly chosen small values of $\gamma$ improve token efficiency over $\gamma=1$ and AdamW under a fixed training protocol.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-shulgin26b, title = {Understanding {MARS}: When Scaling Momentum Correction Provably Helps}, author = {Shulgin, Egor and Gadaev, Tamaz and Khirirat, Sarit and Richt\'{a}rik, Peter}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {112936--112965}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/shulgin26b/shulgin26b.pdf}, url = {https://proceedings.mlr.press/v306/shulgin26b.html}, abstract = {MARS (Yuan et al., 2025) has recently emerged as a strong optimizer for large language model (LLM) training by scaling the correction term in momentum-based variance reduction (MVR). However, existing theory does not explain why this modification can improve convergence over the unscaled MVR choice $\gamma=1$. In this paper, we provide a theoretical explanation for this phenomenon. We introduce $\gamma$-similarity, a refined similarity condition that captures how the scaling coefficient interacts with the stochastic gradient-difference structure. This condition recovers standard similarity at $\gamma=1$ and smoothness at $\gamma=0$. Using $\gamma$-similarity, we derive convergence guarantees for fixed-$\gamma$ MARS whose complexity depends explicitly on $\gamma$ and the corresponding $\gamma$-similarity constant. The bound reveals why small values of $\gamma$ can be beneficial: they may reduce the similarity term enough to outweigh the penalty from deviating from MVR. We prove that optimizing $\gamma$ gives MARS a lower complexity guarantee than MVR. Experiments with MARS-AdamW on GPT-style LLM pretraining corroborate the theory, showing that properly chosen small values of $\gamma$ improve token efficiency over $\gamma=1$ and AdamW under a fixed training protocol.} }
Endnote
%0 Conference Paper %T Understanding MARS: When Scaling Momentum Correction Provably Helps %A Egor Shulgin %A Tamaz Gadaev %A Sarit Khirirat %A Peter Richtárik %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-shulgin26b %I PMLR %P 112936--112965 %U https://proceedings.mlr.press/v306/shulgin26b.html %V 306 %X MARS (Yuan et al., 2025) has recently emerged as a strong optimizer for large language model (LLM) training by scaling the correction term in momentum-based variance reduction (MVR). However, existing theory does not explain why this modification can improve convergence over the unscaled MVR choice $\gamma=1$. In this paper, we provide a theoretical explanation for this phenomenon. We introduce $\gamma$-similarity, a refined similarity condition that captures how the scaling coefficient interacts with the stochastic gradient-difference structure. This condition recovers standard similarity at $\gamma=1$ and smoothness at $\gamma=0$. Using $\gamma$-similarity, we derive convergence guarantees for fixed-$\gamma$ MARS whose complexity depends explicitly on $\gamma$ and the corresponding $\gamma$-similarity constant. The bound reveals why small values of $\gamma$ can be beneficial: they may reduce the similarity term enough to outweigh the penalty from deviating from MVR. We prove that optimizing $\gamma$ gives MARS a lower complexity guarantee than MVR. Experiments with MARS-AdamW on GPT-style LLM pretraining corroborate the theory, showing that properly chosen small values of $\gamma$ improve token efficiency over $\gamma=1$ and AdamW under a fixed training protocol.
APA
Shulgin, E., Gadaev, T., Khirirat, S. & Richtárik, P.. (2026). Understanding MARS: When Scaling Momentum Correction Provably Helps. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:112936-112965 Available from https://proceedings.mlr.press/v306/shulgin26b.html.

Related Material