Personalized Federated Learning using Hypernetworks

Aviv Shamsian, Aviv Navon, Ethan Fetaya, Gal Chechik
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:9489-9502, 2021.

Abstract

Personalized federated learning is tasked with training machine learning models for multiple clients, each with its own data distribution. The goal is to train personalized models collaboratively while accounting for data disparities across clients and reducing communication costs. We propose a novel approach to this problem using hypernetworks, termed pFedHN for personalized Federated HyperNetworks. In this approach, a central hypernetwork model is trained to generate a set of models, one model for each client. This architecture provides effective parameter sharing across clients while maintaining the capacity to generate unique and diverse personal models. Furthermore, since hypernetwork parameters are never transmitted, this approach decouples the communication cost from the trainable model size. We test pFedHN empirically in several personalized federated learning challenges and find that it outperforms previous methods. Finally, since hypernetworks share information across clients, we show that pFedHN can generalize better to new clients whose distributions differ from any client observed during training.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-shamsian21a, title = {Personalized Federated Learning using Hypernetworks}, author = {Shamsian, Aviv and Navon, Aviv and Fetaya, Ethan and Chechik, Gal}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {9489--9502}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/shamsian21a/shamsian21a.pdf}, url = {https://proceedings.mlr.press/v139/shamsian21a.html}, abstract = {Personalized federated learning is tasked with training machine learning models for multiple clients, each with its own data distribution. The goal is to train personalized models collaboratively while accounting for data disparities across clients and reducing communication costs. We propose a novel approach to this problem using hypernetworks, termed pFedHN for personalized Federated HyperNetworks. In this approach, a central hypernetwork model is trained to generate a set of models, one model for each client. This architecture provides effective parameter sharing across clients while maintaining the capacity to generate unique and diverse personal models. Furthermore, since hypernetwork parameters are never transmitted, this approach decouples the communication cost from the trainable model size. We test pFedHN empirically in several personalized federated learning challenges and find that it outperforms previous methods. Finally, since hypernetworks share information across clients, we show that pFedHN can generalize better to new clients whose distributions differ from any client observed during training.} }
Endnote
%0 Conference Paper %T Personalized Federated Learning using Hypernetworks %A Aviv Shamsian %A Aviv Navon %A Ethan Fetaya %A Gal Chechik %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-shamsian21a %I PMLR %P 9489--9502 %U https://proceedings.mlr.press/v139/shamsian21a.html %V 139 %X Personalized federated learning is tasked with training machine learning models for multiple clients, each with its own data distribution. The goal is to train personalized models collaboratively while accounting for data disparities across clients and reducing communication costs. We propose a novel approach to this problem using hypernetworks, termed pFedHN for personalized Federated HyperNetworks. In this approach, a central hypernetwork model is trained to generate a set of models, one model for each client. This architecture provides effective parameter sharing across clients while maintaining the capacity to generate unique and diverse personal models. Furthermore, since hypernetwork parameters are never transmitted, this approach decouples the communication cost from the trainable model size. We test pFedHN empirically in several personalized federated learning challenges and find that it outperforms previous methods. Finally, since hypernetworks share information across clients, we show that pFedHN can generalize better to new clients whose distributions differ from any client observed during training.
APA
Shamsian, A., Navon, A., Fetaya, E. & Chechik, G.. (2021). Personalized Federated Learning using Hypernetworks. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:9489-9502 Available from https://proceedings.mlr.press/v139/shamsian21a.html.

Related Material