COBRA: Contribution-Based Bayesian Rank Allocation for Parameter-Efficient Fine-Tuning

Hongcheng Ding, Xuanze Zhao, Xuanhuang Liu, Jing Jin, Shamsul Nahar Abdullah, Deshinta Arrova Dewi
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:25351-25401, 2026.

Abstract

Full fine-tuning of large language models (LLMs) incurs prohibitive computational and storage costs. Parameter-efficient fine-tuning (PEFT) addresses this limitation, with Low-Rank Adaptation (LoRA) gaining widespread adoption due to its simplicity and zero inference overhead. However, LoRA and its variants typically rely on uniform rank allocation or a single importance metric such as gradient magnitude or output sensitivity to guide rank distribution. This approach fails to recognize that gradient magnitude and output contribution are decoupled properties, leading to suboptimal allocation where critical layers are under-provisioned while less important ones waste capacity. To address this challenge, we propose COBRA, a principled framework integrating dual importance factors for adaptive rank allocation. COBRA operates in three stages: (1) layer conductance attribution quantifies each layer’s contribution via path-integral attribution; (2) dual-factor aggregation combines contribution with adaptation demand, producing the Task-Adaptive Layer Conductance (TA-LC) distribution; and (3) Bayesian rank allocation translates this distribution into optimal heterogeneous ranks via variational optimization. Layer conductance provides layer-level interpretability by explicitly quantifying how much each layer contributes to predictions without redundancy, directly aligning with the granularity of rank allocation decisions and enabling principled cross-layer comparison for rank distribution. Experiments across diverse architectures and tasks demonstrate that COBRA consistently outperforms existing methods, achieving up to 1.6 points improvement on GLUE and a 6.6% average MSE reduction in high-rank regression regimes under comparable parameter budgets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-ding26y, title = {{COBRA}: Contribution-Based {B}ayesian Rank Allocation for Parameter-Efficient Fine-Tuning}, author = {Ding, Hongcheng and Zhao, Xuanze and Liu, Xuanhuang and Jin, Jing and Abdullah, Shamsul Nahar and Dewi, Deshinta Arrova}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {25351--25401}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/ding26y/ding26y.pdf}, url = {https://proceedings.mlr.press/v306/ding26y.html}, abstract = {Full fine-tuning of large language models (LLMs) incurs prohibitive computational and storage costs. Parameter-efficient fine-tuning (PEFT) addresses this limitation, with Low-Rank Adaptation (LoRA) gaining widespread adoption due to its simplicity and zero inference overhead. However, LoRA and its variants typically rely on uniform rank allocation or a single importance metric such as gradient magnitude or output sensitivity to guide rank distribution. This approach fails to recognize that gradient magnitude and output contribution are decoupled properties, leading to suboptimal allocation where critical layers are under-provisioned while less important ones waste capacity. To address this challenge, we propose COBRA, a principled framework integrating dual importance factors for adaptive rank allocation. COBRA operates in three stages: (1) layer conductance attribution quantifies each layer’s contribution via path-integral attribution; (2) dual-factor aggregation combines contribution with adaptation demand, producing the Task-Adaptive Layer Conductance (TA-LC) distribution; and (3) Bayesian rank allocation translates this distribution into optimal heterogeneous ranks via variational optimization. Layer conductance provides layer-level interpretability by explicitly quantifying how much each layer contributes to predictions without redundancy, directly aligning with the granularity of rank allocation decisions and enabling principled cross-layer comparison for rank distribution. Experiments across diverse architectures and tasks demonstrate that COBRA consistently outperforms existing methods, achieving up to 1.6 points improvement on GLUE and a 6.6% average MSE reduction in high-rank regression regimes under comparable parameter budgets.} }
Endnote
%0 Conference Paper %T COBRA: Contribution-Based Bayesian Rank Allocation for Parameter-Efficient Fine-Tuning %A Hongcheng Ding %A Xuanze Zhao %A Xuanhuang Liu %A Jing Jin %A Shamsul Nahar Abdullah %A Deshinta Arrova Dewi %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-ding26y %I PMLR %P 25351--25401 %U https://proceedings.mlr.press/v306/ding26y.html %V 306 %X Full fine-tuning of large language models (LLMs) incurs prohibitive computational and storage costs. Parameter-efficient fine-tuning (PEFT) addresses this limitation, with Low-Rank Adaptation (LoRA) gaining widespread adoption due to its simplicity and zero inference overhead. However, LoRA and its variants typically rely on uniform rank allocation or a single importance metric such as gradient magnitude or output sensitivity to guide rank distribution. This approach fails to recognize that gradient magnitude and output contribution are decoupled properties, leading to suboptimal allocation where critical layers are under-provisioned while less important ones waste capacity. To address this challenge, we propose COBRA, a principled framework integrating dual importance factors for adaptive rank allocation. COBRA operates in three stages: (1) layer conductance attribution quantifies each layer’s contribution via path-integral attribution; (2) dual-factor aggregation combines contribution with adaptation demand, producing the Task-Adaptive Layer Conductance (TA-LC) distribution; and (3) Bayesian rank allocation translates this distribution into optimal heterogeneous ranks via variational optimization. Layer conductance provides layer-level interpretability by explicitly quantifying how much each layer contributes to predictions without redundancy, directly aligning with the granularity of rank allocation decisions and enabling principled cross-layer comparison for rank distribution. Experiments across diverse architectures and tasks demonstrate that COBRA consistently outperforms existing methods, achieving up to 1.6 points improvement on GLUE and a 6.6% average MSE reduction in high-rank regression regimes under comparable parameter budgets.
APA
Ding, H., Zhao, X., Liu, X., Jin, J., Abdullah, S.N. & Dewi, D.A.. (2026). COBRA: Contribution-Based Bayesian Rank Allocation for Parameter-Efficient Fine-Tuning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:25351-25401 Available from https://proceedings.mlr.press/v306/ding26y.html.

Related Material