CGSVD: Cascaded Granular Singular Value Decomposition for Large Language Model Compression

Yuli Chen, Shuhao Zhang, Jiale Han, Fanshen Meng, Haishen Jiang, Bo Cheng, Qiang Tong, Xiulei Liu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:18124-18141, 2026.

Abstract

The exponential growth in the parameter scale of Large Language Models (LLMs) has precipitated an urgent demand for efficient compression techniques to facilitate practical deployment. To address this challenge, low-rank decomposition based on Singular Value Decomposition (SVD) offers a principled, hardware-friendly pathway for compressing LLMs without retraining. However, existing training-free approaches predominantly rely on uniform rank allocation, implicitly assuming homogeneous redundancy across the model depth and thereby neglecting the inherent non-uniformity of representational evolution. To bridge this gap, we introduce CGSVD, a Cascaded Granular Singular Value Decomposition framework that leverages a dual-level non-uniform allocation strategy to maximize semantic preservation. Specifically, we quantify inter-layer significance via angular distance and assess intra-layer compressibility through spectral entropy, enabling precise identification of critical architectural components. Furthermore, we propose an Iterative Residual Filling (IRF) mechanism to bridge the parameter gap caused by integer-rank truncation and ensure strict adherence to global compression targets. Extensive experiments on representative LLM families ranging from 3B to 13B parameters verify the superiority of our approach. Notably, under a 30% compression ratio on the LLaMA3.1-8B model, CGSVD achieves a remarkable average zero-shot accuracy boost of 6.08% and reduces perplexity by 15.46 compared to the baseline. We release the code to facilitate future research.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chen26ge, title = {{CGSVD}: Cascaded Granular Singular Value Decomposition for Large Language Model Compression}, author = {Chen, Yuli and Zhang, Shuhao and Han, Jiale and Meng, Fanshen and Jiang, Haishen and Cheng, Bo and Tong, Qiang and Liu, Xiulei}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {18124--18141}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26ge/chen26ge.pdf}, url = {https://proceedings.mlr.press/v306/chen26ge.html}, abstract = {The exponential growth in the parameter scale of Large Language Models (LLMs) has precipitated an urgent demand for efficient compression techniques to facilitate practical deployment. To address this challenge, low-rank decomposition based on Singular Value Decomposition (SVD) offers a principled, hardware-friendly pathway for compressing LLMs without retraining. However, existing training-free approaches predominantly rely on uniform rank allocation, implicitly assuming homogeneous redundancy across the model depth and thereby neglecting the inherent non-uniformity of representational evolution. To bridge this gap, we introduce CGSVD, a Cascaded Granular Singular Value Decomposition framework that leverages a dual-level non-uniform allocation strategy to maximize semantic preservation. Specifically, we quantify inter-layer significance via angular distance and assess intra-layer compressibility through spectral entropy, enabling precise identification of critical architectural components. Furthermore, we propose an Iterative Residual Filling (IRF) mechanism to bridge the parameter gap caused by integer-rank truncation and ensure strict adherence to global compression targets. Extensive experiments on representative LLM families ranging from 3B to 13B parameters verify the superiority of our approach. Notably, under a 30% compression ratio on the LLaMA3.1-8B model, CGSVD achieves a remarkable average zero-shot accuracy boost of 6.08% and reduces perplexity by 15.46 compared to the baseline. We release the code to facilitate future research.} }
Endnote
%0 Conference Paper %T CGSVD: Cascaded Granular Singular Value Decomposition for Large Language Model Compression %A Yuli Chen %A Shuhao Zhang %A Jiale Han %A Fanshen Meng %A Haishen Jiang %A Bo Cheng %A Qiang Tong %A Xiulei Liu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chen26ge %I PMLR %P 18124--18141 %U https://proceedings.mlr.press/v306/chen26ge.html %V 306 %X The exponential growth in the parameter scale of Large Language Models (LLMs) has precipitated an urgent demand for efficient compression techniques to facilitate practical deployment. To address this challenge, low-rank decomposition based on Singular Value Decomposition (SVD) offers a principled, hardware-friendly pathway for compressing LLMs without retraining. However, existing training-free approaches predominantly rely on uniform rank allocation, implicitly assuming homogeneous redundancy across the model depth and thereby neglecting the inherent non-uniformity of representational evolution. To bridge this gap, we introduce CGSVD, a Cascaded Granular Singular Value Decomposition framework that leverages a dual-level non-uniform allocation strategy to maximize semantic preservation. Specifically, we quantify inter-layer significance via angular distance and assess intra-layer compressibility through spectral entropy, enabling precise identification of critical architectural components. Furthermore, we propose an Iterative Residual Filling (IRF) mechanism to bridge the parameter gap caused by integer-rank truncation and ensure strict adherence to global compression targets. Extensive experiments on representative LLM families ranging from 3B to 13B parameters verify the superiority of our approach. Notably, under a 30% compression ratio on the LLaMA3.1-8B model, CGSVD achieves a remarkable average zero-shot accuracy boost of 6.08% and reduces perplexity by 15.46 compared to the baseline. We release the code to facilitate future research.
APA
Chen, Y., Zhang, S., Han, J., Meng, F., Jiang, H., Cheng, B., Tong, Q. & Liu, X.. (2026). CGSVD: Cascaded Granular Singular Value Decomposition for Large Language Model Compression. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:18124-18141 Available from https://proceedings.mlr.press/v306/chen26ge.html.

Related Material