Collaborative and Efficient Personalization with Mixtures of Adaptors

Abdulla Jasem Almansoori, Samuel Horváth, Martin Takáč
Conference on Parsimony and Learning, PMLR 280:1328-1364, 2025.

Abstract

Heterogenous data is prevalent in real-world federated learning. We propose a parameter-efficient framework, Federated Low-Rank Adaptive Learning (FLoRAL), that allows clients to personalize in groups by mixing between low-rank adaptors, where the mixtures are client-specific. FLoRAL is a model parameterization that casts personalized federated learning as a multi-task learning problem, with weight sharing as an implicit regularizer. It is memory-efficient, as the personalized parameters (i.e., base model + adaptors) are all federated. Our results show that FLoRAL can generalize better than a mixture of full models when data are scarce. It can also consistently personalize better than models with a locally tuned adaptor per client. This demonstrates the benefits of "federated personalization" and its robustness against overfitting. We derive the convergence rates and show theoretically that FLoRAL can lead to better variance reduction of the base model’s gradients.

Cite this Paper

BibTeX
@InProceedings{pmlr-v280-almansoori25a, title = {Collaborative and Efficient Personalization with Mixtures of Adaptors}, author = {Almansoori, Abdulla Jasem and Horv\'{a}th, Samuel and Tak\'{a}\v{c}, Martin}, booktitle = {Conference on Parsimony and Learning}, pages = {1328--1364}, year = {2025}, editor = {Chen, Beidi and Liu, Shijia and Pilanci, Mert and Su, Weijie and Sulam, Jeremias and Wang, Yuxiang and Zhu, Zhihui}, volume = {280}, series = {Proceedings of Machine Learning Research}, month = {24--27 Mar}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v280/main/assets/almansoori25a/almansoori25a.pdf}, url = {https://proceedings.mlr.press/v280/almansoori25a.html}, abstract = {Heterogenous data is prevalent in real-world federated learning. We propose a parameter-efficient framework, Federated Low-Rank Adaptive Learning (FLoRAL), that allows clients to personalize in groups by mixing between low-rank adaptors, where the mixtures are client-specific. FLoRAL is a model parameterization that casts personalized federated learning as a multi-task learning problem, with weight sharing as an implicit regularizer. It is memory-efficient, as the personalized parameters (i.e., base model + adaptors) are all federated. Our results show that FLoRAL can generalize better than a mixture of full models when data are scarce. It can also consistently personalize better than models with a locally tuned adaptor per client. This demonstrates the benefits of "federated personalization" and its robustness against overfitting. We derive the convergence rates and show theoretically that FLoRAL can lead to better variance reduction of the base model’s gradients.} }
Endnote
%0 Conference Paper %T Collaborative and Efficient Personalization with Mixtures of Adaptors %A Abdulla Jasem Almansoori %A Samuel Horváth %A Martin Takáč %B Conference on Parsimony and Learning %C Proceedings of Machine Learning Research %D 2025 %E Beidi Chen %E Shijia Liu %E Mert Pilanci %E Weijie Su %E Jeremias Sulam %E Yuxiang Wang %E Zhihui Zhu %F pmlr-v280-almansoori25a %I PMLR %P 1328--1364 %U https://proceedings.mlr.press/v280/almansoori25a.html %V 280 %X Heterogenous data is prevalent in real-world federated learning. We propose a parameter-efficient framework, Federated Low-Rank Adaptive Learning (FLoRAL), that allows clients to personalize in groups by mixing between low-rank adaptors, where the mixtures are client-specific. FLoRAL is a model parameterization that casts personalized federated learning as a multi-task learning problem, with weight sharing as an implicit regularizer. It is memory-efficient, as the personalized parameters (i.e., base model + adaptors) are all federated. Our results show that FLoRAL can generalize better than a mixture of full models when data are scarce. It can also consistently personalize better than models with a locally tuned adaptor per client. This demonstrates the benefits of "federated personalization" and its robustness against overfitting. We derive the convergence rates and show theoretically that FLoRAL can lead to better variance reduction of the base model’s gradients.
APA
Almansoori, A.J., Horváth, S. & Takáč, M.. (2025). Collaborative and Efficient Personalization with Mixtures of Adaptors. Conference on Parsimony and Learning, in Proceedings of Machine Learning Research 280:1328-1364 Available from https://proceedings.mlr.press/v280/almansoori25a.html.

Related Material