Fine-Tuning of Transformer models with Frames

Harshavardhan Adepu, Li Zhang, Sanjiv Kumar, Vikas Singh
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:544-564, 2026.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scales with the size of the model, $\mathcal{O}(dr)$, where $d$ is the model’s hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $\Delta W$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling highly efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, strongly reducing the memory footprint and parameter count. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames, give sizable compute benefits. Our technical analysis shows that FrameFT allows obtaining formal convergence results. We evaluate our method across a suite of supervised fine-tuning benchmarks, primarily focusing on language tasks, but also report applicability to vision models. Our empirical evaluations show that FrameFT achieves performance on par with or exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters and less memory.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-adepu26a, title = {Fine-Tuning of Transformer models with Frames}, author = {Adepu, Harshavardhan and Zhang, Li and Kumar, Sanjiv and Singh, Vikas}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {544--564}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/adepu26a/adepu26a.pdf}, url = {https://proceedings.mlr.press/v306/adepu26a.html}, abstract = {Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scales with the size of the model, $\mathcal{O}(dr)$, where $d$ is the model’s hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $\Delta W$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling highly efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, strongly reducing the memory footprint and parameter count. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames, give sizable compute benefits. Our technical analysis shows that FrameFT allows obtaining formal convergence results. We evaluate our method across a suite of supervised fine-tuning benchmarks, primarily focusing on language tasks, but also report applicability to vision models. Our empirical evaluations show that FrameFT achieves performance on par with or exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters and less memory.} }
Endnote
%0 Conference Paper %T Fine-Tuning of Transformer models with Frames %A Harshavardhan Adepu %A Li Zhang %A Sanjiv Kumar %A Vikas Singh %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-adepu26a %I PMLR %P 544--564 %U https://proceedings.mlr.press/v306/adepu26a.html %V 306 %X Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scales with the size of the model, $\mathcal{O}(dr)$, where $d$ is the model’s hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $\Delta W$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling highly efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, strongly reducing the memory footprint and parameter count. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames, give sizable compute benefits. Our technical analysis shows that FrameFT allows obtaining formal convergence results. We evaluate our method across a suite of supervised fine-tuning benchmarks, primarily focusing on language tasks, but also report applicability to vision models. Our empirical evaluations show that FrameFT achieves performance on par with or exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters and less memory.
APA
Adepu, H., Zhang, L., Kumar, S. & Singh, V.. (2026). Fine-Tuning of Transformer models with Frames. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:544-564 Available from https://proceedings.mlr.press/v306/adepu26a.html.

Related Material