Diving into Kronecker Adapters: Component Design Matters

Jiayu Bai, Danchen Yu, Zhenyu Liao, Tianqi Hou, Feng Zhou, Robert C Qiu, Zenan Ling
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:5479-5501, 2026.

Abstract

Kronecker adapters have emerged as a promising approach for fine-tuning large-scale models, enabling high-rank updates through tunable component structures. However, existing work largely treats the component structure as a fixed or heuristic design choice, leaving the dimensions and number of Kronecker components underexplored. In this paper, we identify component structure as a key factor governing the capacity of Kronecker adapters. We perform a fine-grained analysis of both the dimensions and number of Kronecker components. In particular, we show that the alignment between Kronecker adapters and full fine-tuning depends on component configurations. Guided by these insights, we propose Component Designed Kronecker Adapters (CDKA). We further provide parameter-budget–aware configuration guidelines and a tailored training stabilization strategy for practical deployment. Experiments across various architectures and modalities demonstrate the effectiveness of CDKA. Code is available at https://github.com/rainstonee/CDKA.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-bai26l, title = {Diving into {K}ronecker Adapters: Component Design Matters}, author = {Bai, Jiayu and Yu, Danchen and Liao, Zhenyu and Hou, Tianqi and Zhou, Feng and Qiu, Robert C and Ling, Zenan}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {5479--5501}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/bai26l/bai26l.pdf}, url = {https://proceedings.mlr.press/v306/bai26l.html}, abstract = {Kronecker adapters have emerged as a promising approach for fine-tuning large-scale models, enabling high-rank updates through tunable component structures. However, existing work largely treats the component structure as a fixed or heuristic design choice, leaving the dimensions and number of Kronecker components underexplored. In this paper, we identify component structure as a key factor governing the capacity of Kronecker adapters. We perform a fine-grained analysis of both the dimensions and number of Kronecker components. In particular, we show that the alignment between Kronecker adapters and full fine-tuning depends on component configurations. Guided by these insights, we propose Component Designed Kronecker Adapters (CDKA). We further provide parameter-budget–aware configuration guidelines and a tailored training stabilization strategy for practical deployment. Experiments across various architectures and modalities demonstrate the effectiveness of CDKA. Code is available at https://github.com/rainstonee/CDKA.} }
Endnote
%0 Conference Paper %T Diving into Kronecker Adapters: Component Design Matters %A Jiayu Bai %A Danchen Yu %A Zhenyu Liao %A Tianqi Hou %A Feng Zhou %A Robert C Qiu %A Zenan Ling %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-bai26l %I PMLR %P 5479--5501 %U https://proceedings.mlr.press/v306/bai26l.html %V 306 %X Kronecker adapters have emerged as a promising approach for fine-tuning large-scale models, enabling high-rank updates through tunable component structures. However, existing work largely treats the component structure as a fixed or heuristic design choice, leaving the dimensions and number of Kronecker components underexplored. In this paper, we identify component structure as a key factor governing the capacity of Kronecker adapters. We perform a fine-grained analysis of both the dimensions and number of Kronecker components. In particular, we show that the alignment between Kronecker adapters and full fine-tuning depends on component configurations. Guided by these insights, we propose Component Designed Kronecker Adapters (CDKA). We further provide parameter-budget–aware configuration guidelines and a tailored training stabilization strategy for practical deployment. Experiments across various architectures and modalities demonstrate the effectiveness of CDKA. Code is available at https://github.com/rainstonee/CDKA.
APA
Bai, J., Yu, D., Liao, Z., Hou, T., Zhou, F., Qiu, R.C. & Ling, Z.. (2026). Diving into Kronecker Adapters: Component Design Matters. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:5479-5501 Available from https://proceedings.mlr.press/v306/bai26l.html.

Related Material