Federated Learning with Partial Model Personalization

Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, Lin Xiao
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:17716-17758, 2022.

Abstract

We consider two federated learning algorithms for training partially personalized models, where the shared and personal parameters are updated either simultaneously or alternately on the devices. Both algorithms have been proposed in the literature, but their convergence properties are not fully understood, especially for the alternating variant. We provide convergence analyses of both algorithms in the general nonconvex setting with partial participation and delineate the regime where one dominates the other. Our experiments on real-world image, text, and speech datasets demonstrate that (a) partial personalization can obtain most of the benefits of full model personalization with a small fraction of personal parameters, and, (b) the alternating update algorithm outperforms the simultaneous update algorithm by a small but consistent margin.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-pillutla22a, title = {Federated Learning with Partial Model Personalization}, author = {Pillutla, Krishna and Malik, Kshitiz and Mohamed, Abdel-Rahman and Rabbat, Mike and Sanjabi, Maziar and Xiao, Lin}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {17716--17758}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/pillutla22a/pillutla22a.pdf}, url = {https://proceedings.mlr.press/v162/pillutla22a.html}, abstract = {We consider two federated learning algorithms for training partially personalized models, where the shared and personal parameters are updated either simultaneously or alternately on the devices. Both algorithms have been proposed in the literature, but their convergence properties are not fully understood, especially for the alternating variant. We provide convergence analyses of both algorithms in the general nonconvex setting with partial participation and delineate the regime where one dominates the other. Our experiments on real-world image, text, and speech datasets demonstrate that (a) partial personalization can obtain most of the benefits of full model personalization with a small fraction of personal parameters, and, (b) the alternating update algorithm outperforms the simultaneous update algorithm by a small but consistent margin.} }
Endnote
%0 Conference Paper %T Federated Learning with Partial Model Personalization %A Krishna Pillutla %A Kshitiz Malik %A Abdel-Rahman Mohamed %A Mike Rabbat %A Maziar Sanjabi %A Lin Xiao %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-pillutla22a %I PMLR %P 17716--17758 %U https://proceedings.mlr.press/v162/pillutla22a.html %V 162 %X We consider two federated learning algorithms for training partially personalized models, where the shared and personal parameters are updated either simultaneously or alternately on the devices. Both algorithms have been proposed in the literature, but their convergence properties are not fully understood, especially for the alternating variant. We provide convergence analyses of both algorithms in the general nonconvex setting with partial participation and delineate the regime where one dominates the other. Our experiments on real-world image, text, and speech datasets demonstrate that (a) partial personalization can obtain most of the benefits of full model personalization with a small fraction of personal parameters, and, (b) the alternating update algorithm outperforms the simultaneous update algorithm by a small but consistent margin.
APA
Pillutla, K., Malik, K., Mohamed, A., Rabbat, M., Sanjabi, M. & Xiao, L.. (2022). Federated Learning with Partial Model Personalization. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:17716-17758 Available from https://proceedings.mlr.press/v162/pillutla22a.html.

Related Material