PFedAtt: Attention-based Personalized Federated Learning on Heterogeneous Clients

Zichen Ma, Yu Lu, Wenye Li, Jinfeng Yi, Shuguang Cui
Proceedings of The 13th Asian Conference on Machine Learning, PMLR 157:1253-1268, 2021.

Abstract

In federated learning, heterogeneity among the clients’ local datasets results in large variations in the number of local updates performed by each client in a communication round. Simply aggregating such local models into a global model will confine the capacity of the system, that is, the single global model will be restricted from delivering good performance on each client’s task. This paper provides a general framework to analyze the convergence of personalized federated learning algorithms. It subsumes previously proposed methods and provides a principled understanding of the computational guarantees. Using insights from this analysis, we propose PFedAtt, a personalized federated learning method that incorporates attention-based grouping to facilitate similar clients’ collaborations. Theoretically, we provide the convergence guarantee for the algorithm, and empirical experiments corroborate the competitive performance of PFedAtt on heterogeneous clients.

Cite this Paper

BibTeX
@InProceedings{pmlr-v157-ma21a, title = {PFedAtt: Attention-based Personalized Federated Learning on Heterogeneous Clients}, author = {Ma, Zichen and Lu, Yu and Li, Wenye and Yi, Jinfeng and Cui, Shuguang}, booktitle = {Proceedings of The 13th Asian Conference on Machine Learning}, pages = {1253--1268}, year = {2021}, editor = {Balasubramanian, Vineeth N. and Tsang, Ivor}, volume = {157}, series = {Proceedings of Machine Learning Research}, month = {17--19 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v157/ma21a/ma21a.pdf}, url = {https://proceedings.mlr.press/v157/ma21a.html}, abstract = {In federated learning, heterogeneity among the clients’ local datasets results in large variations in the number of local updates performed by each client in a communication round. Simply aggregating such local models into a global model will confine the capacity of the system, that is, the single global model will be restricted from delivering good performance on each client’s task. This paper provides a general framework to analyze the convergence of personalized federated learning algorithms. It subsumes previously proposed methods and provides a principled understanding of the computational guarantees. Using insights from this analysis, we propose PFedAtt, a personalized federated learning method that incorporates attention-based grouping to facilitate similar clients’ collaborations. Theoretically, we provide the convergence guarantee for the algorithm, and empirical experiments corroborate the competitive performance of PFedAtt on heterogeneous clients.} }
Endnote
%0 Conference Paper %T PFedAtt: Attention-based Personalized Federated Learning on Heterogeneous Clients %A Zichen Ma %A Yu Lu %A Wenye Li %A Jinfeng Yi %A Shuguang Cui %B Proceedings of The 13th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Vineeth N. Balasubramanian %E Ivor Tsang %F pmlr-v157-ma21a %I PMLR %P 1253--1268 %U https://proceedings.mlr.press/v157/ma21a.html %V 157 %X In federated learning, heterogeneity among the clients’ local datasets results in large variations in the number of local updates performed by each client in a communication round. Simply aggregating such local models into a global model will confine the capacity of the system, that is, the single global model will be restricted from delivering good performance on each client’s task. This paper provides a general framework to analyze the convergence of personalized federated learning algorithms. It subsumes previously proposed methods and provides a principled understanding of the computational guarantees. Using insights from this analysis, we propose PFedAtt, a personalized federated learning method that incorporates attention-based grouping to facilitate similar clients’ collaborations. Theoretically, we provide the convergence guarantee for the algorithm, and empirical experiments corroborate the competitive performance of PFedAtt on heterogeneous clients.
APA
Ma, Z., Lu, Y., Li, W., Yi, J. & Cui, S.. (2021). PFedAtt: Attention-based Personalized Federated Learning on Heterogeneous Clients. Proceedings of The 13th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 157:1253-1268 Available from https://proceedings.mlr.press/v157/ma21a.html.

Related Material