LBI-FL: Low-Bit Integerized Federated Learning with Temporally Dynamic Bit-Width Allocation

Li Ding, Hao Zhang, Wenrui Dai, Chenglin Li, Weijia Lu, Zhifei Yang, Xiaodong Zhang, Xiaofeng Ma, Junni Zou, Hongkai Xiong
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:13885-13899, 2025.

Abstract

Federated learning (FL) is greatly challenged by the communication bottleneck and computation limitation on clients. Existing methods based on quantization for FL cannot simultaneously reduce the uplink and downlink communication cost and mitigate the computation burden on clients. To address this problem, in this paper, we propose the first low-bit integerized federated learning (LBI-FL) framework that quantizes the weights, activations, and gradients to lower than INT8 precision to evidently reduce the communication and computational costs. Specifically, we achieve dynamical temporal bit-width allocation for weights, activations, and gradients along the training trajectory via reinforcement learning. An agent is trained to determine bit-width allocation by comprehensively considering the states like current bit-width, training stage, and quantization loss as the state. The agent efficiently trained on small-scale datasets can be well generalized to train varying network architectures on non-independent and identically distributed datasets. Furthermore, we demonstrated in theory that federated learning with gradient quantization achieves an equivalent convergence rate to FedAvg. The proposed LBI-FL can reduce the communication costs by 8 times compared to full-precision FL. Extensive experiments show that the proposed LBI-FL achieves a reduction of more than 50% BitOPs per client on average for FL with less than 2% accuracy loss compared to low-bit training with INT8 precision.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-ding25e, title = {{LBI}-{FL}: Low-Bit Integerized Federated Learning with Temporally Dynamic Bit-Width Allocation}, author = {Ding, Li and Zhang, Hao and Dai, Wenrui and Li, Chenglin and Lu, Weijia and Yang, Zhifei and Zhang, Xiaodong and Ma, Xiaofeng and Zou, Junni and Xiong, Hongkai}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {13885--13899}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/ding25e/ding25e.pdf}, url = {https://proceedings.mlr.press/v267/ding25e.html}, abstract = {Federated learning (FL) is greatly challenged by the communication bottleneck and computation limitation on clients. Existing methods based on quantization for FL cannot simultaneously reduce the uplink and downlink communication cost and mitigate the computation burden on clients. To address this problem, in this paper, we propose the first low-bit integerized federated learning (LBI-FL) framework that quantizes the weights, activations, and gradients to lower than INT8 precision to evidently reduce the communication and computational costs. Specifically, we achieve dynamical temporal bit-width allocation for weights, activations, and gradients along the training trajectory via reinforcement learning. An agent is trained to determine bit-width allocation by comprehensively considering the states like current bit-width, training stage, and quantization loss as the state. The agent efficiently trained on small-scale datasets can be well generalized to train varying network architectures on non-independent and identically distributed datasets. Furthermore, we demonstrated in theory that federated learning with gradient quantization achieves an equivalent convergence rate to FedAvg. The proposed LBI-FL can reduce the communication costs by 8 times compared to full-precision FL. Extensive experiments show that the proposed LBI-FL achieves a reduction of more than 50% BitOPs per client on average for FL with less than 2% accuracy loss compared to low-bit training with INT8 precision.} }
Endnote
%0 Conference Paper %T LBI-FL: Low-Bit Integerized Federated Learning with Temporally Dynamic Bit-Width Allocation %A Li Ding %A Hao Zhang %A Wenrui Dai %A Chenglin Li %A Weijia Lu %A Zhifei Yang %A Xiaodong Zhang %A Xiaofeng Ma %A Junni Zou %A Hongkai Xiong %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-ding25e %I PMLR %P 13885--13899 %U https://proceedings.mlr.press/v267/ding25e.html %V 267 %X Federated learning (FL) is greatly challenged by the communication bottleneck and computation limitation on clients. Existing methods based on quantization for FL cannot simultaneously reduce the uplink and downlink communication cost and mitigate the computation burden on clients. To address this problem, in this paper, we propose the first low-bit integerized federated learning (LBI-FL) framework that quantizes the weights, activations, and gradients to lower than INT8 precision to evidently reduce the communication and computational costs. Specifically, we achieve dynamical temporal bit-width allocation for weights, activations, and gradients along the training trajectory via reinforcement learning. An agent is trained to determine bit-width allocation by comprehensively considering the states like current bit-width, training stage, and quantization loss as the state. The agent efficiently trained on small-scale datasets can be well generalized to train varying network architectures on non-independent and identically distributed datasets. Furthermore, we demonstrated in theory that federated learning with gradient quantization achieves an equivalent convergence rate to FedAvg. The proposed LBI-FL can reduce the communication costs by 8 times compared to full-precision FL. Extensive experiments show that the proposed LBI-FL achieves a reduction of more than 50% BitOPs per client on average for FL with less than 2% accuracy loss compared to low-bit training with INT8 precision.
APA
Ding, L., Zhang, H., Dai, W., Li, C., Lu, W., Yang, Z., Zhang, X., Ma, X., Zou, J. & Xiong, H.. (2025). LBI-FL: Low-Bit Integerized Federated Learning with Temporally Dynamic Bit-Width Allocation. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:13885-13899 Available from https://proceedings.mlr.press/v267/ding25e.html.

Related Material