Federated Averaging Langevin Dynamics: Toward a unified theory and new algorithms

Vincent Plassier, Eric Moulines, Alain Durmus
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:5299-5356, 2023.

Abstract

This paper focuses on Bayesian inference in a federated learning context (FL). While several distributed MCMC algorithms have been proposed, few consider the specific limitations of FL such as communication bottlenecks and statistical heterogeneity. Recently, Federated Averaging Langevin Dynamics (FALD) was introduced, which extends the Federated Averaging algorithm to Bayesian inference. We obtain a novel tight non-asymptotic upper bound on the Wasserstein distance to the global posterior for FALD. This bound highlights the effects of statistical heterogeneity, which causes a drift in the local updates that negatively impacts convergence. We propose a new algorithm VR-FALD* that uses control variates to correct the client drift. We establish non-asymptotic bounds showing that VR-FALD* is not affected by statistical heterogeneity. Finally, we illustrate our results on several FL benchmarks for Bayesian inference.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-plassier23a, title = {Federated Averaging Langevin Dynamics: Toward a unified theory and new algorithms}, author = {Plassier, Vincent and Moulines, Eric and Durmus, Alain}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {5299--5356}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/plassier23a/plassier23a.pdf}, url = {https://proceedings.mlr.press/v206/plassier23a.html}, abstract = {This paper focuses on Bayesian inference in a federated learning context (FL). While several distributed MCMC algorithms have been proposed, few consider the specific limitations of FL such as communication bottlenecks and statistical heterogeneity. Recently, Federated Averaging Langevin Dynamics (FALD) was introduced, which extends the Federated Averaging algorithm to Bayesian inference. We obtain a novel tight non-asymptotic upper bound on the Wasserstein distance to the global posterior for FALD. This bound highlights the effects of statistical heterogeneity, which causes a drift in the local updates that negatively impacts convergence. We propose a new algorithm VR-FALD* that uses control variates to correct the client drift. We establish non-asymptotic bounds showing that VR-FALD* is not affected by statistical heterogeneity. Finally, we illustrate our results on several FL benchmarks for Bayesian inference.} }
Endnote
%0 Conference Paper %T Federated Averaging Langevin Dynamics: Toward a unified theory and new algorithms %A Vincent Plassier %A Eric Moulines %A Alain Durmus %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-plassier23a %I PMLR %P 5299--5356 %U https://proceedings.mlr.press/v206/plassier23a.html %V 206 %X This paper focuses on Bayesian inference in a federated learning context (FL). While several distributed MCMC algorithms have been proposed, few consider the specific limitations of FL such as communication bottlenecks and statistical heterogeneity. Recently, Federated Averaging Langevin Dynamics (FALD) was introduced, which extends the Federated Averaging algorithm to Bayesian inference. We obtain a novel tight non-asymptotic upper bound on the Wasserstein distance to the global posterior for FALD. This bound highlights the effects of statistical heterogeneity, which causes a drift in the local updates that negatively impacts convergence. We propose a new algorithm VR-FALD* that uses control variates to correct the client drift. We establish non-asymptotic bounds showing that VR-FALD* is not affected by statistical heterogeneity. Finally, we illustrate our results on several FL benchmarks for Bayesian inference.
APA
Plassier, V., Moulines, E. & Durmus, A.. (2023). Federated Averaging Langevin Dynamics: Toward a unified theory and new algorithms. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:5299-5356 Available from https://proceedings.mlr.press/v206/plassier23a.html.

Related Material