KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity

Damien Lesens, Beheshteh T. Rakhshan, Guillaume Rabusseau
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:3556-3564, 2026.

Abstract

The Key–Value (KV) cache is central to the efficiency of transformer-based large language models (LLMs), storing previously computed vectors to accelerate inference. Yet, as sequence length and batch size grow, the cache becomes a major memory bottleneck. Prior compression methods typically apply low-rank decomposition to keys alone or attempt to jointly embed queries and keys, but both approaches neglect that attention fundamentally depends on their inner products. In this work, we prove that such strategies are sub-optimal for approximating the attention matrix. We introduce KQ-SVD, a simple and computationally efficient method that directly performs an optimal low-rank decomposition of the attention matrix via a closed-form solution. By targeting the true source of redundancy, KQ-SVD preserves attention outputs with higher fidelity under compression. Extensive evaluations on LLaMA and Mistral models demonstrate that our approach consistently delivers superior projection quality.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-lesens26a, title = { KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity }, author = {Lesens, Damien and Rakhshan, Beheshteh T. and Rabusseau, Guillaume}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {3556--3564}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/lesens26a/lesens26a.pdf}, url = {https://proceedings.mlr.press/v300/lesens26a.html}, abstract = { The Key–Value (KV) cache is central to the efficiency of transformer-based large language models (LLMs), storing previously computed vectors to accelerate inference. Yet, as sequence length and batch size grow, the cache becomes a major memory bottleneck. Prior compression methods typically apply low-rank decomposition to keys alone or attempt to jointly embed queries and keys, but both approaches neglect that attention fundamentally depends on their inner products. In this work, we prove that such strategies are sub-optimal for approximating the attention matrix. We introduce KQ-SVD, a simple and computationally efficient method that directly performs an optimal low-rank decomposition of the attention matrix via a closed-form solution. By targeting the true source of redundancy, KQ-SVD preserves attention outputs with higher fidelity under compression. Extensive evaluations on LLaMA and Mistral models demonstrate that our approach consistently delivers superior projection quality. } }
Endnote
%0 Conference Paper %T KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity %A Damien Lesens %A Beheshteh T. Rakhshan %A Guillaume Rabusseau %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-lesens26a %I PMLR %P 3556--3564 %U https://proceedings.mlr.press/v300/lesens26a.html %V 300 %X The Key–Value (KV) cache is central to the efficiency of transformer-based large language models (LLMs), storing previously computed vectors to accelerate inference. Yet, as sequence length and batch size grow, the cache becomes a major memory bottleneck. Prior compression methods typically apply low-rank decomposition to keys alone or attempt to jointly embed queries and keys, but both approaches neglect that attention fundamentally depends on their inner products. In this work, we prove that such strategies are sub-optimal for approximating the attention matrix. We introduce KQ-SVD, a simple and computationally efficient method that directly performs an optimal low-rank decomposition of the attention matrix via a closed-form solution. By targeting the true source of redundancy, KQ-SVD preserves attention outputs with higher fidelity under compression. Extensive evaluations on LLaMA and Mistral models demonstrate that our approach consistently delivers superior projection quality.
APA
Lesens, D., Rakhshan, B.T. & Rabusseau, G.. (2026). KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:3556-3564 Available from https://proceedings.mlr.press/v300/lesens26a.html.

Related Material