Probabilistic Fusion of Neural Networks that Incorporates Global Information

Peng Xiao, Biao Zhang, Samuel Cheng, Ke Wei, Shuqin Zhang
Proceedings of The 14th Asian Conference on Machine Learning, PMLR 189:1149-1164, 2023.

Abstract

As one of the approaches in Federated Learning, model fusion distills models trained on local clients into a global model. The previous method, Probabilistic Federated Neural Matching (PFNM), can match and fuse local neural networks with varying global model sizes and data heterogeneity using the Bayesian nonparametric framework. However, the alternating optimization process applied by PFNM causes absence of global neuron information. In this paper, we propose a new method that extends PFNM by introducing a Kullback-Leibler (KL) divergence penalty, so that it can exploit information in both local and global neurons. We show theoretically that the extended PFNM with a penalty derived from KL divergence can fix the drawback of PFNM by making a balance between Euclidean distance and the prior probability of neurons. Experiments on deep fully-connected as well as deep convolutional neural networks demonstrate that our new method outperforms popular state-of-the-art federated learning methods in both image classification and semantic segmentation tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v189-xiao23b, title = {Probabilistic Fusion of Neural Networks that Incorporates Global Information}, author = {Xiao, Peng and Zhang, Biao and Cheng, Samuel and Wei, Ke and Zhang, Shuqin}, booktitle = {Proceedings of The 14th Asian Conference on Machine Learning}, pages = {1149--1164}, year = {2023}, editor = {Khan, Emtiyaz and Gonen, Mehmet}, volume = {189}, series = {Proceedings of Machine Learning Research}, month = {12--14 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v189/xiao23b/xiao23b.pdf}, url = {https://proceedings.mlr.press/v189/xiao23b.html}, abstract = {As one of the approaches in Federated Learning, model fusion distills models trained on local clients into a global model. The previous method, Probabilistic Federated Neural Matching (PFNM), can match and fuse local neural networks with varying global model sizes and data heterogeneity using the Bayesian nonparametric framework. However, the alternating optimization process applied by PFNM causes absence of global neuron information. In this paper, we propose a new method that extends PFNM by introducing a Kullback-Leibler (KL) divergence penalty, so that it can exploit information in both local and global neurons. We show theoretically that the extended PFNM with a penalty derived from KL divergence can fix the drawback of PFNM by making a balance between Euclidean distance and the prior probability of neurons. Experiments on deep fully-connected as well as deep convolutional neural networks demonstrate that our new method outperforms popular state-of-the-art federated learning methods in both image classification and semantic segmentation tasks.} }
Endnote
%0 Conference Paper %T Probabilistic Fusion of Neural Networks that Incorporates Global Information %A Peng Xiao %A Biao Zhang %A Samuel Cheng %A Ke Wei %A Shuqin Zhang %B Proceedings of The 14th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Emtiyaz Khan %E Mehmet Gonen %F pmlr-v189-xiao23b %I PMLR %P 1149--1164 %U https://proceedings.mlr.press/v189/xiao23b.html %V 189 %X As one of the approaches in Federated Learning, model fusion distills models trained on local clients into a global model. The previous method, Probabilistic Federated Neural Matching (PFNM), can match and fuse local neural networks with varying global model sizes and data heterogeneity using the Bayesian nonparametric framework. However, the alternating optimization process applied by PFNM causes absence of global neuron information. In this paper, we propose a new method that extends PFNM by introducing a Kullback-Leibler (KL) divergence penalty, so that it can exploit information in both local and global neurons. We show theoretically that the extended PFNM with a penalty derived from KL divergence can fix the drawback of PFNM by making a balance between Euclidean distance and the prior probability of neurons. Experiments on deep fully-connected as well as deep convolutional neural networks demonstrate that our new method outperforms popular state-of-the-art federated learning methods in both image classification and semantic segmentation tasks.
APA
Xiao, P., Zhang, B., Cheng, S., Wei, K. & Zhang, S.. (2023). Probabilistic Fusion of Neural Networks that Incorporates Global Information. Proceedings of The 14th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 189:1149-1164 Available from https://proceedings.mlr.press/v189/xiao23b.html.

Related Material