Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression

Ali Abbasi, Chayne Thrash, Haoran Qin, Shansita Sharma, Sepehr Seifi, Soheil Kolouri
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:84-100, 2026.

Abstract

Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD-based compression reduces storage and can speed up inference via low-rank factors, yet performance depends on how rank is allocated under a global compression ratio. Prior methods often use homogeneous ranks for similarly sized matrices, despite large differences in loss sensitivity, or rely on expensive iterative pre-truncation optimization to determine per matrix ranks. We propose Zero Sum SVD (ZS-SVD), a post-training method that performs global singular component selection using activation whitening and first-order calibration loss estimates in whitened coordinates. ZS-SVD prunes components across the whole model with a zero sum rule that keeps the cumulative predicted loss change near zero, automatically yielding heterogeneous ranks without solving a rank allocation optimization. Motivated by evidence that gradients near pretrained solutions exhibit low rank structure, we also introduce an optional lightweight correction that applies a single projected gradient update after truncation, followed by re-truncation. Extensive experiments across multiple LLM architectures show consistent gains across diverse benchmarks and compression ratios. Code is available at https://github.com/mint-vu/Zero-Sum-SVD

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-abbasi26a, title = {Zero Sum {SVD}: Balancing Loss Sensitivity for Low Rank {LLM} Compression}, author = {Abbasi, Ali and Thrash, Chayne and Qin, Haoran and Sharma, Shansita and Seifi, Sepehr and Kolouri, Soheil}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {84--100}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/abbasi26a/abbasi26a.pdf}, url = {https://proceedings.mlr.press/v306/abbasi26a.html}, abstract = {Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD-based compression reduces storage and can speed up inference via low-rank factors, yet performance depends on how rank is allocated under a global compression ratio. Prior methods often use homogeneous ranks for similarly sized matrices, despite large differences in loss sensitivity, or rely on expensive iterative pre-truncation optimization to determine per matrix ranks. We propose Zero Sum SVD (ZS-SVD), a post-training method that performs global singular component selection using activation whitening and first-order calibration loss estimates in whitened coordinates. ZS-SVD prunes components across the whole model with a zero sum rule that keeps the cumulative predicted loss change near zero, automatically yielding heterogeneous ranks without solving a rank allocation optimization. Motivated by evidence that gradients near pretrained solutions exhibit low rank structure, we also introduce an optional lightweight correction that applies a single projected gradient update after truncation, followed by re-truncation. Extensive experiments across multiple LLM architectures show consistent gains across diverse benchmarks and compression ratios. Code is available at https://github.com/mint-vu/Zero-Sum-SVD} }
Endnote
%0 Conference Paper %T Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression %A Ali Abbasi %A Chayne Thrash %A Haoran Qin %A Shansita Sharma %A Sepehr Seifi %A Soheil Kolouri %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-abbasi26a %I PMLR %P 84--100 %U https://proceedings.mlr.press/v306/abbasi26a.html %V 306 %X Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD-based compression reduces storage and can speed up inference via low-rank factors, yet performance depends on how rank is allocated under a global compression ratio. Prior methods often use homogeneous ranks for similarly sized matrices, despite large differences in loss sensitivity, or rely on expensive iterative pre-truncation optimization to determine per matrix ranks. We propose Zero Sum SVD (ZS-SVD), a post-training method that performs global singular component selection using activation whitening and first-order calibration loss estimates in whitened coordinates. ZS-SVD prunes components across the whole model with a zero sum rule that keeps the cumulative predicted loss change near zero, automatically yielding heterogeneous ranks without solving a rank allocation optimization. Motivated by evidence that gradients near pretrained solutions exhibit low rank structure, we also introduce an optional lightweight correction that applies a single projected gradient update after truncation, followed by re-truncation. Extensive experiments across multiple LLM architectures show consistent gains across diverse benchmarks and compression ratios. Code is available at https://github.com/mint-vu/Zero-Sum-SVD
APA
Abbasi, A., Thrash, C., Qin, H., Sharma, S., Seifi, S. & Kolouri, S.. (2026). Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:84-100 Available from https://proceedings.mlr.press/v306/abbasi26a.html.

Related Material