Asynchronous Distributed Learning : Adapting to Gradient Delays without Prior Knowledge

Rotem Zamir Aviv, Ido Hakimi, Assaf Schuster, Kfir Yehuda Levy
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:436-445, 2021.

Abstract

We consider stochastic convex optimization problems, where several machines act asynchronously in parallel while sharing a common memory. We propose a robust training method for the constrained setting and derive non asymptotic convergence guarantees that do not depend on prior knowledge of update delays, objective smoothness, and gradient variance. Conversely, existing methods for this setting crucially rely on this prior knowledge, which render them unsuitable for essentially all shared-resources computational environments, such as clouds and data centers. Concretely, existing approaches are unable to accommodate changes in the delays which result from dynamic allocation of the machines, while our method implicitly adapts to such changes.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-aviv21a, title = {Asynchronous Distributed Learning : Adapting to Gradient Delays without Prior Knowledge}, author = {Aviv, Rotem Zamir and Hakimi, Ido and Schuster, Assaf and Levy, Kfir Yehuda}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {436--445}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/aviv21a/aviv21a.pdf}, url = {https://proceedings.mlr.press/v139/aviv21a.html}, abstract = {We consider stochastic convex optimization problems, where several machines act asynchronously in parallel while sharing a common memory. We propose a robust training method for the constrained setting and derive non asymptotic convergence guarantees that do not depend on prior knowledge of update delays, objective smoothness, and gradient variance. Conversely, existing methods for this setting crucially rely on this prior knowledge, which render them unsuitable for essentially all shared-resources computational environments, such as clouds and data centers. Concretely, existing approaches are unable to accommodate changes in the delays which result from dynamic allocation of the machines, while our method implicitly adapts to such changes.} }
Endnote
%0 Conference Paper %T Asynchronous Distributed Learning : Adapting to Gradient Delays without Prior Knowledge %A Rotem Zamir Aviv %A Ido Hakimi %A Assaf Schuster %A Kfir Yehuda Levy %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-aviv21a %I PMLR %P 436--445 %U https://proceedings.mlr.press/v139/aviv21a.html %V 139 %X We consider stochastic convex optimization problems, where several machines act asynchronously in parallel while sharing a common memory. We propose a robust training method for the constrained setting and derive non asymptotic convergence guarantees that do not depend on prior knowledge of update delays, objective smoothness, and gradient variance. Conversely, existing methods for this setting crucially rely on this prior knowledge, which render them unsuitable for essentially all shared-resources computational environments, such as clouds and data centers. Concretely, existing approaches are unable to accommodate changes in the delays which result from dynamic allocation of the machines, while our method implicitly adapts to such changes.
APA
Aviv, R.Z., Hakimi, I., Schuster, A. & Levy, K.Y.. (2021). Asynchronous Distributed Learning : Adapting to Gradient Delays without Prior Knowledge. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:436-445 Available from https://proceedings.mlr.press/v139/aviv21a.html.

Related Material