FedHera: Towards Drift-Resilient Federated Fine-tuning with Heterogeneous Resources

Ke Xiao, Qiyuan Wang, Christos Anagnostopoulos, Zhuoran Tan, Wenhao Li
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:138978-139003, 2026.

Abstract

Driven by the imperative to leverage privacy-sensitive data scattered across decentralized devices, federated fine-tuning has emerged as a vital paradigm for adapting large language models without compromising data privacy. Yet, its practical efficacy is bottlenecked by severe client resource heterogeneity. Existing truncation-based methods typically couple the transmitted rank with the trainable rank, which (i) under-utilizes bandwidth on communication-rich but compute-limited clients and (ii) exacerbates truncation-induced gradient drift. To address this, we propose FedHera, a resource-decoupled framework that explicitly differentiates information reception from gradient optimization. FedHera employs a spectrum-preserving allocation strategy to maximize the transfer of global knowledge (via high-rank singular values) within bandwidth limits, irrespective of training constraints. Furthermore, we introduce a prefix-gating mechanism that utilizes the downloaded high-capacity basis as a frozen reference to guide local updates, thereby minimizing the optimization gap caused by aggressive truncation. Extensive experiments under different heterogeneous settings show that FedHera improves stability and accuracy over state-of-the-art baselines.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-xiao26p, title = {{F}ed{H}era: Towards Drift-Resilient Federated Fine-tuning with Heterogeneous Resources}, author = {Xiao, Ke and Wang, Qiyuan and Anagnostopoulos, Christos and Tan, Zhuoran and Li, Wenhao}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {138978--139003}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/xiao26p/xiao26p.pdf}, url = {https://proceedings.mlr.press/v306/xiao26p.html}, abstract = {Driven by the imperative to leverage privacy-sensitive data scattered across decentralized devices, federated fine-tuning has emerged as a vital paradigm for adapting large language models without compromising data privacy. Yet, its practical efficacy is bottlenecked by severe client resource heterogeneity. Existing truncation-based methods typically couple the transmitted rank with the trainable rank, which (i) under-utilizes bandwidth on communication-rich but compute-limited clients and (ii) exacerbates truncation-induced gradient drift. To address this, we propose FedHera, a resource-decoupled framework that explicitly differentiates information reception from gradient optimization. FedHera employs a spectrum-preserving allocation strategy to maximize the transfer of global knowledge (via high-rank singular values) within bandwidth limits, irrespective of training constraints. Furthermore, we introduce a prefix-gating mechanism that utilizes the downloaded high-capacity basis as a frozen reference to guide local updates, thereby minimizing the optimization gap caused by aggressive truncation. Extensive experiments under different heterogeneous settings show that FedHera improves stability and accuracy over state-of-the-art baselines.} }
Endnote
%0 Conference Paper %T FedHera: Towards Drift-Resilient Federated Fine-tuning with Heterogeneous Resources %A Ke Xiao %A Qiyuan Wang %A Christos Anagnostopoulos %A Zhuoran Tan %A Wenhao Li %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-xiao26p %I PMLR %P 138978--139003 %U https://proceedings.mlr.press/v306/xiao26p.html %V 306 %X Driven by the imperative to leverage privacy-sensitive data scattered across decentralized devices, federated fine-tuning has emerged as a vital paradigm for adapting large language models without compromising data privacy. Yet, its practical efficacy is bottlenecked by severe client resource heterogeneity. Existing truncation-based methods typically couple the transmitted rank with the trainable rank, which (i) under-utilizes bandwidth on communication-rich but compute-limited clients and (ii) exacerbates truncation-induced gradient drift. To address this, we propose FedHera, a resource-decoupled framework that explicitly differentiates information reception from gradient optimization. FedHera employs a spectrum-preserving allocation strategy to maximize the transfer of global knowledge (via high-rank singular values) within bandwidth limits, irrespective of training constraints. Furthermore, we introduce a prefix-gating mechanism that utilizes the downloaded high-capacity basis as a frozen reference to guide local updates, thereby minimizing the optimization gap caused by aggressive truncation. Extensive experiments under different heterogeneous settings show that FedHera improves stability and accuracy over state-of-the-art baselines.
APA
Xiao, K., Wang, Q., Anagnostopoulos, C., Tan, Z. & Li, W.. (2026). FedHera: Towards Drift-Resilient Federated Fine-tuning with Heterogeneous Resources. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:138978-139003 Available from https://proceedings.mlr.press/v306/xiao26p.html.

Related Material