FedPeWS: Personalized Warmup via Subnetworks for Enhanced Heterogeneous Federated Learning

Nurbek Tastan, Samuel Horváth, Martin Takáč, Karthik Nandakumar
Conference on Parsimony and Learning, PMLR 280:462-483, 2025.

Abstract

Statistical data heterogeneity is a significant barrier to convergence in federated learning (FL). While prior work has advanced heterogeneous FL through better optimization objectives, these methods fall short when there is *extreme* data heterogeneity among collaborating participants. We hypothesize that convergence under extreme data heterogeneity is primarily hindered due to the aggregation of conflicting updates from the participants in the initial collaboration rounds. To overcome this problem, we propose a warmup phase where each participant learns a personalized mask and updates only a subnetwork of the full model. This *personalized warmup* allows the participants to focus initially on learning specific *subnetworks* tailored to the heterogeneity of their data. After the warmup phase, the participants revert to standard federated optimization, where all parameters are communicated. We empirically demonstrate that the proposed personalized warmup via subnetworks (*FedPeWS*) approach improves accuracy and convergence speed over standard federated optimization methods. The code can be found at https://github.com/tnurbek/fedpews.

Cite this Paper

BibTeX
@InProceedings{pmlr-v280-tastan25a, title = {FedPeWS: Personalized Warmup via Subnetworks for Enhanced Heterogeneous Federated Learning}, author = {Tastan, Nurbek and Horv\'{a}th, Samuel and Tak\'{a}\v{c}, Martin and Nandakumar, Karthik}, booktitle = {Conference on Parsimony and Learning}, pages = {462--483}, year = {2025}, editor = {Chen, Beidi and Liu, Shijia and Pilanci, Mert and Su, Weijie and Sulam, Jeremias and Wang, Yuxiang and Zhu, Zhihui}, volume = {280}, series = {Proceedings of Machine Learning Research}, month = {24--27 Mar}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v280/main/assets/tastan25a/tastan25a.pdf}, url = {https://proceedings.mlr.press/v280/tastan25a.html}, abstract = {Statistical data heterogeneity is a significant barrier to convergence in federated learning (FL). While prior work has advanced heterogeneous FL through better optimization objectives, these methods fall short when there is *extreme* data heterogeneity among collaborating participants. We hypothesize that convergence under extreme data heterogeneity is primarily hindered due to the aggregation of conflicting updates from the participants in the initial collaboration rounds. To overcome this problem, we propose a warmup phase where each participant learns a personalized mask and updates only a subnetwork of the full model. This *personalized warmup* allows the participants to focus initially on learning specific *subnetworks* tailored to the heterogeneity of their data. After the warmup phase, the participants revert to standard federated optimization, where all parameters are communicated. We empirically demonstrate that the proposed personalized warmup via subnetworks (*FedPeWS*) approach improves accuracy and convergence speed over standard federated optimization methods. The code can be found at https://github.com/tnurbek/fedpews.} }
Endnote
%0 Conference Paper %T FedPeWS: Personalized Warmup via Subnetworks for Enhanced Heterogeneous Federated Learning %A Nurbek Tastan %A Samuel Horváth %A Martin Takáč %A Karthik Nandakumar %B Conference on Parsimony and Learning %C Proceedings of Machine Learning Research %D 2025 %E Beidi Chen %E Shijia Liu %E Mert Pilanci %E Weijie Su %E Jeremias Sulam %E Yuxiang Wang %E Zhihui Zhu %F pmlr-v280-tastan25a %I PMLR %P 462--483 %U https://proceedings.mlr.press/v280/tastan25a.html %V 280 %X Statistical data heterogeneity is a significant barrier to convergence in federated learning (FL). While prior work has advanced heterogeneous FL through better optimization objectives, these methods fall short when there is *extreme* data heterogeneity among collaborating participants. We hypothesize that convergence under extreme data heterogeneity is primarily hindered due to the aggregation of conflicting updates from the participants in the initial collaboration rounds. To overcome this problem, we propose a warmup phase where each participant learns a personalized mask and updates only a subnetwork of the full model. This *personalized warmup* allows the participants to focus initially on learning specific *subnetworks* tailored to the heterogeneity of their data. After the warmup phase, the participants revert to standard federated optimization, where all parameters are communicated. We empirically demonstrate that the proposed personalized warmup via subnetworks (*FedPeWS*) approach improves accuracy and convergence speed over standard federated optimization methods. The code can be found at https://github.com/tnurbek/fedpews.
APA
Tastan, N., Horváth, S., Takáč, M. & Nandakumar, K.. (2025). FedPeWS: Personalized Warmup via Subnetworks for Enhanced Heterogeneous Federated Learning. Conference on Parsimony and Learning, in Proceedings of Machine Learning Research 280:462-483 Available from https://proceedings.mlr.press/v280/tastan25a.html.

Related Material