CSPLoRA: Confidence-Guided Structural Planning for Low-Rank Adaptation

Huiming Ding, Xiaochen Li, Jianhui Ma, Xu An, Yihui Yang, Zhenyu Tan
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:24997-25017, 2026.

Abstract

Low-Rank Adaptation (LoRA) has become the de facto paradigm for parameter-efficient fine-tuning, with its effectiveness critically influenced by rank allocation across modules. However, existing approaches face a fundamental dilemma: uniform allocation ignores module heterogeneity, while adaptive methods introduce expensive training overhead or lack reusability across configurations. We propose CSPLoRA (Confidence-guided Structural Planning for LoRA), a decoupled framework that reweights probe samples by prediction uncertainty to obtain more discriminative module importance estimates. The key insight is that hard samples—those the model struggles with—provide more informative gradient signals for identifying critical modules than easy samples. For a fixed task-model pair, the resulting structural priors can be reused across compatible rank budgets and LoRA backends, supporting a practical "probe once, deploy everywhere" workflow. Experiments on GLUE, commonsense reasoning, and arithmetic tasks show that CSPLoRA improves over uniform LoRA on average (+1.25 points on LLaMA-2-7B commonsense reasoning) while maintaining comparable parameters, with the planned rank structure reusable across compatible LoRA variants.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-ding26h, title = {{CSPL}o{RA}: Confidence-Guided Structural Planning for Low-Rank Adaptation}, author = {Ding, Huiming and Li, Xiaochen and Ma, Jianhui and An, Xu and Yang, Yihui and Tan, Zhenyu}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {24997--25017}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/ding26h/ding26h.pdf}, url = {https://proceedings.mlr.press/v306/ding26h.html}, abstract = {Low-Rank Adaptation (LoRA) has become the de facto paradigm for parameter-efficient fine-tuning, with its effectiveness critically influenced by rank allocation across modules. However, existing approaches face a fundamental dilemma: uniform allocation ignores module heterogeneity, while adaptive methods introduce expensive training overhead or lack reusability across configurations. We propose CSPLoRA (Confidence-guided Structural Planning for LoRA), a decoupled framework that reweights probe samples by prediction uncertainty to obtain more discriminative module importance estimates. The key insight is that hard samples—those the model struggles with—provide more informative gradient signals for identifying critical modules than easy samples. For a fixed task-model pair, the resulting structural priors can be reused across compatible rank budgets and LoRA backends, supporting a practical "probe once, deploy everywhere" workflow. Experiments on GLUE, commonsense reasoning, and arithmetic tasks show that CSPLoRA improves over uniform LoRA on average (+1.25 points on LLaMA-2-7B commonsense reasoning) while maintaining comparable parameters, with the planned rank structure reusable across compatible LoRA variants.} }
Endnote
%0 Conference Paper %T CSPLoRA: Confidence-Guided Structural Planning for Low-Rank Adaptation %A Huiming Ding %A Xiaochen Li %A Jianhui Ma %A Xu An %A Yihui Yang %A Zhenyu Tan %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-ding26h %I PMLR %P 24997--25017 %U https://proceedings.mlr.press/v306/ding26h.html %V 306 %X Low-Rank Adaptation (LoRA) has become the de facto paradigm for parameter-efficient fine-tuning, with its effectiveness critically influenced by rank allocation across modules. However, existing approaches face a fundamental dilemma: uniform allocation ignores module heterogeneity, while adaptive methods introduce expensive training overhead or lack reusability across configurations. We propose CSPLoRA (Confidence-guided Structural Planning for LoRA), a decoupled framework that reweights probe samples by prediction uncertainty to obtain more discriminative module importance estimates. The key insight is that hard samples—those the model struggles with—provide more informative gradient signals for identifying critical modules than easy samples. For a fixed task-model pair, the resulting structural priors can be reused across compatible rank budgets and LoRA backends, supporting a practical "probe once, deploy everywhere" workflow. Experiments on GLUE, commonsense reasoning, and arithmetic tasks show that CSPLoRA improves over uniform LoRA on average (+1.25 points on LLaMA-2-7B commonsense reasoning) while maintaining comparable parameters, with the planned rank structure reusable across compatible LoRA variants.
APA
Ding, H., Li, X., Ma, J., An, X., Yang, Y. & Tan, Z.. (2026). CSPLoRA: Confidence-Guided Structural Planning for Low-Rank Adaptation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:24997-25017 Available from https://proceedings.mlr.press/v306/ding26h.html.

Related Material