FedEMoE: Improving Personalization on Heterogeneous Federated Learning via Elastic Mixture of Experts Architecture

Haizhou Du, Lixin Huang, Zonghan Wu, Huan Huo
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:26454-26479, 2026.

Abstract

Heterogeneous federated learning (HtFL) has emerged as a promising approach to address heterogeneity in local computational resources and data distribution. However, existing methods cause performance degradation of model personalization because personalized and generalized knowledge are either intertwined or dominated by one of them. To address this issue, we propose a novel Elastic Mixture of Experts (EMoE) architecture on HtFL, namely FedEMoE, decoupling personalization from generalization. Specially, FedEMoE employs a multi-scale feature extraction mechanism via personalized experts to enrich personalized knowledge. Furthermore, we design an elastic shared expert to break the transferred knowledge bottleneck across heterogeneous client models. The elastic shared expert can adaptively expand or shrink according to the status of each expert by the weight spectrum analysis, respectively. Extensive experiments across statistical and model heterogeneity settings demonstrate that FedEMoE significantly outperforms state-of-the art methods on the accuracy of each heterogeneous model over diverse datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-du26d, title = {{F}ed{EM}o{E}: Improving Personalization on Heterogeneous Federated Learning via Elastic Mixture of Experts Architecture}, author = {Du, Haizhou and Huang, Lixin and Wu, Zonghan and Huo, Huan}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {26454--26479}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/du26d/du26d.pdf}, url = {https://proceedings.mlr.press/v306/du26d.html}, abstract = {Heterogeneous federated learning (HtFL) has emerged as a promising approach to address heterogeneity in local computational resources and data distribution. However, existing methods cause performance degradation of model personalization because personalized and generalized knowledge are either intertwined or dominated by one of them. To address this issue, we propose a novel Elastic Mixture of Experts (EMoE) architecture on HtFL, namely FedEMoE, decoupling personalization from generalization. Specially, FedEMoE employs a multi-scale feature extraction mechanism via personalized experts to enrich personalized knowledge. Furthermore, we design an elastic shared expert to break the transferred knowledge bottleneck across heterogeneous client models. The elastic shared expert can adaptively expand or shrink according to the status of each expert by the weight spectrum analysis, respectively. Extensive experiments across statistical and model heterogeneity settings demonstrate that FedEMoE significantly outperforms state-of-the art methods on the accuracy of each heterogeneous model over diverse datasets.} }
Endnote
%0 Conference Paper %T FedEMoE: Improving Personalization on Heterogeneous Federated Learning via Elastic Mixture of Experts Architecture %A Haizhou Du %A Lixin Huang %A Zonghan Wu %A Huan Huo %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-du26d %I PMLR %P 26454--26479 %U https://proceedings.mlr.press/v306/du26d.html %V 306 %X Heterogeneous federated learning (HtFL) has emerged as a promising approach to address heterogeneity in local computational resources and data distribution. However, existing methods cause performance degradation of model personalization because personalized and generalized knowledge are either intertwined or dominated by one of them. To address this issue, we propose a novel Elastic Mixture of Experts (EMoE) architecture on HtFL, namely FedEMoE, decoupling personalization from generalization. Specially, FedEMoE employs a multi-scale feature extraction mechanism via personalized experts to enrich personalized knowledge. Furthermore, we design an elastic shared expert to break the transferred knowledge bottleneck across heterogeneous client models. The elastic shared expert can adaptively expand or shrink according to the status of each expert by the weight spectrum analysis, respectively. Extensive experiments across statistical and model heterogeneity settings demonstrate that FedEMoE significantly outperforms state-of-the art methods on the accuracy of each heterogeneous model over diverse datasets.
APA
Du, H., Huang, L., Wu, Z. & Huo, H.. (2026). FedEMoE: Improving Personalization on Heterogeneous Federated Learning via Elastic Mixture of Experts Architecture. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:26454-26479 Available from https://proceedings.mlr.press/v306/du26d.html.

Related Material