Accelerating Federated Learning with Quick Distributed Mean Estimation

Ran Ben-Basat, Shay Vargaftik, Amit Portnoy, Gil Einziger, Yaniv Ben-Itzhak, Michael Mitzenmacher
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:3410-3442, 2024.

Abstract

Distributed Mean Estimation (DME), in which $n$ clients communicate vectors to a parameter server that estimates their average, is a fundamental building block in communication-efficient federated learning. In this paper, we improve on previous DME techniques that achieve the optimal $O(1/n)$ Normalized Mean Squared Error (NMSE) guarantee by asymptotically improving the complexity for either encoding or decoding (or both). To achieve this, we formalize the problem in a novel way that allows us to use off-the-shelf mathematical solvers to design the quantization. Using various datasets and training tasks, we demonstrate how QUIC-FL achieves state of the art accuracy with faster encoding and decoding times compared to other DME methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-ben-basat24a, title = {Accelerating Federated Learning with Quick Distributed Mean Estimation}, author = {Ben-Basat, Ran and Vargaftik, Shay and Portnoy, Amit and Einziger, Gil and Ben-Itzhak, Yaniv and Mitzenmacher, Michael}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {3410--3442}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/ben-basat24a/ben-basat24a.pdf}, url = {https://proceedings.mlr.press/v235/ben-basat24a.html}, abstract = {Distributed Mean Estimation (DME), in which $n$ clients communicate vectors to a parameter server that estimates their average, is a fundamental building block in communication-efficient federated learning. In this paper, we improve on previous DME techniques that achieve the optimal $O(1/n)$ Normalized Mean Squared Error (NMSE) guarantee by asymptotically improving the complexity for either encoding or decoding (or both). To achieve this, we formalize the problem in a novel way that allows us to use off-the-shelf mathematical solvers to design the quantization. Using various datasets and training tasks, we demonstrate how QUIC-FL achieves state of the art accuracy with faster encoding and decoding times compared to other DME methods.} }
Endnote
%0 Conference Paper %T Accelerating Federated Learning with Quick Distributed Mean Estimation %A Ran Ben-Basat %A Shay Vargaftik %A Amit Portnoy %A Gil Einziger %A Yaniv Ben-Itzhak %A Michael Mitzenmacher %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-ben-basat24a %I PMLR %P 3410--3442 %U https://proceedings.mlr.press/v235/ben-basat24a.html %V 235 %X Distributed Mean Estimation (DME), in which $n$ clients communicate vectors to a parameter server that estimates their average, is a fundamental building block in communication-efficient federated learning. In this paper, we improve on previous DME techniques that achieve the optimal $O(1/n)$ Normalized Mean Squared Error (NMSE) guarantee by asymptotically improving the complexity for either encoding or decoding (or both). To achieve this, we formalize the problem in a novel way that allows us to use off-the-shelf mathematical solvers to design the quantization. Using various datasets and training tasks, we demonstrate how QUIC-FL achieves state of the art accuracy with faster encoding and decoding times compared to other DME methods.
APA
Ben-Basat, R., Vargaftik, S., Portnoy, A., Einziger, G., Ben-Itzhak, Y. & Mitzenmacher, M.. (2024). Accelerating Federated Learning with Quick Distributed Mean Estimation. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:3410-3442 Available from https://proceedings.mlr.press/v235/ben-basat24a.html.

Related Material