Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling

Qiao Xiao, Boqian Wu, Patrik Okanovic, Tomasz Sternal, Maurice Van Keulen, Elena Mocanu, Mykola Pechenizkiy, Decebal Constantin Mocanu, Torsten Hoefler
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:139069-139085, 2026.

Abstract

Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates. In this work, we show that the naive use of standard Adam-based optimizers leads to a cold-start issue for newly regrown parameters, resulting in excessively large updates and disrupted training dynamics. To address this issue, we propose Sparse Memory-Efficient Training (SMET), which stabilizes DST with optimizer warm-up and improves training progress through density-aware learning-rate scaling. SMET further reduces memory consumption by storing gradients and optimizer states only for active parameters. We provide a theoretical analysis of the update behaviors under SMET, showing improved optimization stability. Extensive experiments demonstrate that SMET enables stable, scalable, and memory-efficient sparse pre-training of LLMs, paving the way for sparse training as a practical alternative to dense training. Our code is publicly available at: https://github.com/QiaoXiao7282/SMET.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-xiao26t, title = {Memory-Efficient {LLM} Training with Dynamic Sparsity: From Stability to Practical Scaling}, author = {Xiao, Qiao and Wu, Boqian and Okanovic, Patrik and Sternal, Tomasz and Keulen, Maurice Van and Mocanu, Elena and Pechenizkiy, Mykola and Mocanu, Decebal Constantin and Hoefler, Torsten}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {139069--139085}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/xiao26t/xiao26t.pdf}, url = {https://proceedings.mlr.press/v306/xiao26t.html}, abstract = {Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates. In this work, we show that the naive use of standard Adam-based optimizers leads to a cold-start issue for newly regrown parameters, resulting in excessively large updates and disrupted training dynamics. To address this issue, we propose Sparse Memory-Efficient Training (SMET), which stabilizes DST with optimizer warm-up and improves training progress through density-aware learning-rate scaling. SMET further reduces memory consumption by storing gradients and optimizer states only for active parameters. We provide a theoretical analysis of the update behaviors under SMET, showing improved optimization stability. Extensive experiments demonstrate that SMET enables stable, scalable, and memory-efficient sparse pre-training of LLMs, paving the way for sparse training as a practical alternative to dense training. Our code is publicly available at: https://github.com/QiaoXiao7282/SMET.} }
Endnote
%0 Conference Paper %T Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling %A Qiao Xiao %A Boqian Wu %A Patrik Okanovic %A Tomasz Sternal %A Maurice Van Keulen %A Elena Mocanu %A Mykola Pechenizkiy %A Decebal Constantin Mocanu %A Torsten Hoefler %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-xiao26t %I PMLR %P 139069--139085 %U https://proceedings.mlr.press/v306/xiao26t.html %V 306 %X Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates. In this work, we show that the naive use of standard Adam-based optimizers leads to a cold-start issue for newly regrown parameters, resulting in excessively large updates and disrupted training dynamics. To address this issue, we propose Sparse Memory-Efficient Training (SMET), which stabilizes DST with optimizer warm-up and improves training progress through density-aware learning-rate scaling. SMET further reduces memory consumption by storing gradients and optimizer states only for active parameters. We provide a theoretical analysis of the update behaviors under SMET, showing improved optimization stability. Extensive experiments demonstrate that SMET enables stable, scalable, and memory-efficient sparse pre-training of LLMs, paving the way for sparse training as a practical alternative to dense training. Our code is publicly available at: https://github.com/QiaoXiao7282/SMET.
APA
Xiao, Q., Wu, B., Okanovic, P., Sternal, T., Keulen, M.V., Mocanu, E., Pechenizkiy, M., Mocanu, D.C. & Hoefler, T.. (2026). Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:139069-139085 Available from https://proceedings.mlr.press/v306/xiao26t.html.

Related Material