FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

Jieming Bian, Lei Wang, Letian Zhang, Jie Xu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:8204-8234, 2026.

Abstract

Federated Learning (FL) with Low-Rank Adaptation (LoRA) has become a standard for privacy-preserving LLM fine-tuning. However, existing personalized methods predominantly operated under a restrictive Flat-Model Assumption: they addressed client-side statistical heterogeneity but treated the model as a monolithic block, ignoring the functional heterogeneity across LLM layers. We argue that these two statistical (horizontal) and functional (vertical) dimensions, are orthogonal in source yet coupled in interaction, implying that the optimal depth of parameter sharing is functionally dependent on client similarity. To address this, we propose FedTreeLoRA, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment. By dynamically constructing an aggregation hierarchy, FedTreeLoRA allows clients to share broad consensus on shallow ’trunks‘ while progressively specializing on deep ‘branches’. Experiments on NLU and NLG benchmarks demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods by effectively reconciling generalization and personalization.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-bian26e, title = {{F}ed{T}ree{L}o{RA}: Reconciling Statistical and Functional Heterogeneity in Federated {L}o{RA} Fine-Tuning}, author = {Bian, Jieming and Wang, Lei and Zhang, Letian and Xu, Jie}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {8204--8234}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/bian26e/bian26e.pdf}, url = {https://proceedings.mlr.press/v306/bian26e.html}, abstract = {Federated Learning (FL) with Low-Rank Adaptation (LoRA) has become a standard for privacy-preserving LLM fine-tuning. However, existing personalized methods predominantly operated under a restrictive Flat-Model Assumption: they addressed client-side statistical heterogeneity but treated the model as a monolithic block, ignoring the functional heterogeneity across LLM layers. We argue that these two statistical (horizontal) and functional (vertical) dimensions, are orthogonal in source yet coupled in interaction, implying that the optimal depth of parameter sharing is functionally dependent on client similarity. To address this, we propose FedTreeLoRA, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment. By dynamically constructing an aggregation hierarchy, FedTreeLoRA allows clients to share broad consensus on shallow ’trunks‘ while progressively specializing on deep ‘branches’. Experiments on NLU and NLG benchmarks demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods by effectively reconciling generalization and personalization.} }
Endnote
%0 Conference Paper %T FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning %A Jieming Bian %A Lei Wang %A Letian Zhang %A Jie Xu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-bian26e %I PMLR %P 8204--8234 %U https://proceedings.mlr.press/v306/bian26e.html %V 306 %X Federated Learning (FL) with Low-Rank Adaptation (LoRA) has become a standard for privacy-preserving LLM fine-tuning. However, existing personalized methods predominantly operated under a restrictive Flat-Model Assumption: they addressed client-side statistical heterogeneity but treated the model as a monolithic block, ignoring the functional heterogeneity across LLM layers. We argue that these two statistical (horizontal) and functional (vertical) dimensions, are orthogonal in source yet coupled in interaction, implying that the optimal depth of parameter sharing is functionally dependent on client similarity. To address this, we propose FedTreeLoRA, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment. By dynamically constructing an aggregation hierarchy, FedTreeLoRA allows clients to share broad consensus on shallow ’trunks‘ while progressively specializing on deep ‘branches’. Experiments on NLU and NLG benchmarks demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods by effectively reconciling generalization and personalization.
APA
Bian, J., Wang, L., Zhang, L. & Xu, J.. (2026). FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:8204-8234 Available from https://proceedings.mlr.press/v306/bian26e.html.

Related Material