LENA: Communication-Efficient Distributed Learning with Self-Triggered Gradient Uploads

Hossein Shokri Ghadikolaei, Sebastian Stich, Martin Jaggi
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:3943-3951, 2021.

Abstract

In distributed optimization, parameter updates from the gradient computing node devices have to be aggregated in every iteration on the orchestrating server. When these updates are sent over an arbitrary commodity network, bandwidth and latency can be limiting factors. We propose a communication framework where nodes may skip unnecessary uploads. Every node locally accumulates an error vector in memory and self-triggers the upload of the memory contents to the parameter server using a significance filter. The server then uses a history of the nodes’ gradients to update the parameter. We characterize the convergence rate of our algorithm in smooth settings (strongly-convex, convex, and non-convex) and show that it enjoys the same convergence rate as when sending gradients every iteration, with substantially fewer uploads. Numerical experiments on real data indicate a significant reduction of used network resources (total communicated bits and latency), especially in large networks, compared to state-of-the-art algorithms. Our results provide important practical insights for using machine learning over resource-constrained networks, including Internet-of-Things and geo-separated datasets across the globe.

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-shokri-ghadikolaei21a, title = { LENA: Communication-Efficient Distributed Learning with Self-Triggered Gradient Uploads }, author = {Shokri Ghadikolaei, Hossein and Stich, Sebastian and Jaggi, Martin}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {3943--3951}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/shokri-ghadikolaei21a/shokri-ghadikolaei21a.pdf}, url = {https://proceedings.mlr.press/v130/shokri-ghadikolaei21a.html}, abstract = { In distributed optimization, parameter updates from the gradient computing node devices have to be aggregated in every iteration on the orchestrating server. When these updates are sent over an arbitrary commodity network, bandwidth and latency can be limiting factors. We propose a communication framework where nodes may skip unnecessary uploads. Every node locally accumulates an error vector in memory and self-triggers the upload of the memory contents to the parameter server using a significance filter. The server then uses a history of the nodes’ gradients to update the parameter. We characterize the convergence rate of our algorithm in smooth settings (strongly-convex, convex, and non-convex) and show that it enjoys the same convergence rate as when sending gradients every iteration, with substantially fewer uploads. Numerical experiments on real data indicate a significant reduction of used network resources (total communicated bits and latency), especially in large networks, compared to state-of-the-art algorithms. Our results provide important practical insights for using machine learning over resource-constrained networks, including Internet-of-Things and geo-separated datasets across the globe. } }
Endnote
%0 Conference Paper %T LENA: Communication-Efficient Distributed Learning with Self-Triggered Gradient Uploads %A Hossein Shokri Ghadikolaei %A Sebastian Stich %A Martin Jaggi %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-shokri-ghadikolaei21a %I PMLR %P 3943--3951 %U https://proceedings.mlr.press/v130/shokri-ghadikolaei21a.html %V 130 %X In distributed optimization, parameter updates from the gradient computing node devices have to be aggregated in every iteration on the orchestrating server. When these updates are sent over an arbitrary commodity network, bandwidth and latency can be limiting factors. We propose a communication framework where nodes may skip unnecessary uploads. Every node locally accumulates an error vector in memory and self-triggers the upload of the memory contents to the parameter server using a significance filter. The server then uses a history of the nodes’ gradients to update the parameter. We characterize the convergence rate of our algorithm in smooth settings (strongly-convex, convex, and non-convex) and show that it enjoys the same convergence rate as when sending gradients every iteration, with substantially fewer uploads. Numerical experiments on real data indicate a significant reduction of used network resources (total communicated bits and latency), especially in large networks, compared to state-of-the-art algorithms. Our results provide important practical insights for using machine learning over resource-constrained networks, including Internet-of-Things and geo-separated datasets across the globe.
APA
Shokri Ghadikolaei, H., Stich, S. & Jaggi, M.. (2021). LENA: Communication-Efficient Distributed Learning with Self-Triggered Gradient Uploads . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:3943-3951 Available from https://proceedings.mlr.press/v130/shokri-ghadikolaei21a.html.

Related Material