Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers

Samuel Erickson, Mikael Johansson
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:28166-28186, 2026.

Abstract

In modern machine learning, parallelization of training is an important strategy for increasing scale. Asynchronous stochastic gradient descent (ASGD), which maximally utilizes available hardware, avoids having to wait for slow workers. However, with constant step sizes, the convergence of ASGD is nonetheless negatively effected by slow workers due to large delays in updates. At the same time, it has been empirically observed in asynchronous training of deep learning models that gradient clipping “stabilizes” training. In this work, we provide a theoretical justification for this behavior, as we show that clipping removes the dependence of the maximum delay in the oracle complexity. We employ a sub-Weibull model of gradient noise which generalize sub-Gaussian and sub-exponential disitributions to more heavy-tailed distributions, motivated by empirical observations in deep learning. We show convergence in expectation, and for the first time in asynchronous optimization, convergence with high probability.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-erickson26a, title = {Clipping Makes Distributed and Federated Asynchronous {SGD} Robust to Stragglers}, author = {Erickson, Samuel and Johansson, Mikael}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {28166--28186}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/erickson26a/erickson26a.pdf}, url = {https://proceedings.mlr.press/v306/erickson26a.html}, abstract = {In modern machine learning, parallelization of training is an important strategy for increasing scale. Asynchronous stochastic gradient descent (ASGD), which maximally utilizes available hardware, avoids having to wait for slow workers. However, with constant step sizes, the convergence of ASGD is nonetheless negatively effected by slow workers due to large delays in updates. At the same time, it has been empirically observed in asynchronous training of deep learning models that gradient clipping “stabilizes” training. In this work, we provide a theoretical justification for this behavior, as we show that clipping removes the dependence of the maximum delay in the oracle complexity. We employ a sub-Weibull model of gradient noise which generalize sub-Gaussian and sub-exponential disitributions to more heavy-tailed distributions, motivated by empirical observations in deep learning. We show convergence in expectation, and for the first time in asynchronous optimization, convergence with high probability.} }
Endnote
%0 Conference Paper %T Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers %A Samuel Erickson %A Mikael Johansson %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-erickson26a %I PMLR %P 28166--28186 %U https://proceedings.mlr.press/v306/erickson26a.html %V 306 %X In modern machine learning, parallelization of training is an important strategy for increasing scale. Asynchronous stochastic gradient descent (ASGD), which maximally utilizes available hardware, avoids having to wait for slow workers. However, with constant step sizes, the convergence of ASGD is nonetheless negatively effected by slow workers due to large delays in updates. At the same time, it has been empirically observed in asynchronous training of deep learning models that gradient clipping “stabilizes” training. In this work, we provide a theoretical justification for this behavior, as we show that clipping removes the dependence of the maximum delay in the oracle complexity. We employ a sub-Weibull model of gradient noise which generalize sub-Gaussian and sub-exponential disitributions to more heavy-tailed distributions, motivated by empirical observations in deep learning. We show convergence in expectation, and for the first time in asynchronous optimization, convergence with high probability.
APA
Erickson, S. & Johansson, M.. (2026). Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:28166-28186 Available from https://proceedings.mlr.press/v306/erickson26a.html.

Related Material