GiVA: Gradient-Informed Bases for Vector-Based Adaptation

Neeraj Gangwar, Rishabh Deshmukh, Michael Shavlovsky, Hancao Li, Vivek Mittal, Lexing Ying, Nickvash Kani
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:4015-4023, 2026.

Abstract

As model sizes continue to grow, parameter-efficient fine-tuning has emerged as a powerful alternative to full fine-tuning. While LoRA is widely adopted among these methods, recent research has explored vector-based adaptation methods due to their extreme parameter efficiency. However, these methods typically require substantially higher ranks than LoRA to match its performance, leading to increased training costs. This work introduces GiVA, a gradient-based initialization strategy for vector-based adaptation. It achieves training times comparable to LoRA and maintains the extreme parameter efficiency of vector-based adaptation. We evaluate GiVA across diverse benchmarks, including natural language understanding, natural language generation, and image classification. Experiments show that our approach consistently outperforms or achieves performance competitive with existing vector-based adaptation methods and LoRA while reducing rank requirements by a factor of eight ($8\times$).

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-gangwar26b, title = { GiVA: Gradient-Informed Bases for Vector-Based Adaptation }, author = {Gangwar, Neeraj and Deshmukh, Rishabh and Shavlovsky, Michael and Li, Hancao and Mittal, Vivek and Ying, Lexing and Kani, Nickvash}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {4015--4023}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/gangwar26b/gangwar26b.pdf}, url = {https://proceedings.mlr.press/v300/gangwar26b.html}, abstract = { As model sizes continue to grow, parameter-efficient fine-tuning has emerged as a powerful alternative to full fine-tuning. While LoRA is widely adopted among these methods, recent research has explored vector-based adaptation methods due to their extreme parameter efficiency. However, these methods typically require substantially higher ranks than LoRA to match its performance, leading to increased training costs. This work introduces GiVA, a gradient-based initialization strategy for vector-based adaptation. It achieves training times comparable to LoRA and maintains the extreme parameter efficiency of vector-based adaptation. We evaluate GiVA across diverse benchmarks, including natural language understanding, natural language generation, and image classification. Experiments show that our approach consistently outperforms or achieves performance competitive with existing vector-based adaptation methods and LoRA while reducing rank requirements by a factor of eight ($8\times$). } }
Endnote
%0 Conference Paper %T GiVA: Gradient-Informed Bases for Vector-Based Adaptation %A Neeraj Gangwar %A Rishabh Deshmukh %A Michael Shavlovsky %A Hancao Li %A Vivek Mittal %A Lexing Ying %A Nickvash Kani %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-gangwar26b %I PMLR %P 4015--4023 %U https://proceedings.mlr.press/v300/gangwar26b.html %V 300 %X As model sizes continue to grow, parameter-efficient fine-tuning has emerged as a powerful alternative to full fine-tuning. While LoRA is widely adopted among these methods, recent research has explored vector-based adaptation methods due to their extreme parameter efficiency. However, these methods typically require substantially higher ranks than LoRA to match its performance, leading to increased training costs. This work introduces GiVA, a gradient-based initialization strategy for vector-based adaptation. It achieves training times comparable to LoRA and maintains the extreme parameter efficiency of vector-based adaptation. We evaluate GiVA across diverse benchmarks, including natural language understanding, natural language generation, and image classification. Experiments show that our approach consistently outperforms or achieves performance competitive with existing vector-based adaptation methods and LoRA while reducing rank requirements by a factor of eight ($8\times$).
APA
Gangwar, N., Deshmukh, R., Shavlovsky, M., Li, H., Mittal, V., Ying, L. & Kani, N.. (2026). GiVA: Gradient-Informed Bases for Vector-Based Adaptation . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:4015-4023 Available from https://proceedings.mlr.press/v300/gangwar26b.html.

Related Material