Advancing SVD-based LLM Compression via Layer-Wise Error Model Search

Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler, Emad Aghajanzadeh, Manoj Rohit Vemparala, Christos Anagnostopoulos, Pierpaolo Mori, Nael Fasfous, Alexander Frickenstein, Daniel Mueller-Gritschneder, Ulf Schlichtmann
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:120620-120653, 2026.

Abstract

Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs. However, existing methods face two key limitations: (i) global rank allocation, where uncalibrated error proxies fail to capture complex error propagation, and (ii) decomposition quality, where Fisher-based estimators suffer from severe rank collapse. In this work, we address these limitations by introducing Layer-wise Error Modeling Search (LEMS) and KFAC-SVD. LEMS advances rank allocation by introducing a layer-wise error surrogate that integrates local and global layer importance alongside a propagation bias, enabling effective global rank allocation via an ILP formulation. KFAC-SVD improves decomposition quality by utilizing token-wise statistics, mitigating the rank deficiency observed in prior Fisher-based SVD approaches. Across Mistral, Qwen3, and Llama3 model families, we show that LEMS consistently outperforms existing search strategies, delivering significant zero-shot accuracy gains of up to 4.8 p.p. that generalize to model sizes of 70B parameters, while KFAC-SVD achieves an average perplexity improvement of 15%. Project Page & Code: https://lems-svd.github.io

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-thoma26a, title = {Advancing {SVD}-based {LLM} Compression via Layer-Wise Error Model Search}, author = {Thoma, Moritz and Groezinger, Maximilian and Forstenh\"{a}usler, Maximilian and Aghajanzadeh, Emad and Vemparala, Manoj Rohit and Anagnostopoulos, Christos and Mori, Pierpaolo and Fasfous, Nael and Frickenstein, Alexander and Mueller-Gritschneder, Daniel and Schlichtmann, Ulf}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {120620--120653}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/thoma26a/thoma26a.pdf}, url = {https://proceedings.mlr.press/v306/thoma26a.html}, abstract = {Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs. However, existing methods face two key limitations: (i) global rank allocation, where uncalibrated error proxies fail to capture complex error propagation, and (ii) decomposition quality, where Fisher-based estimators suffer from severe rank collapse. In this work, we address these limitations by introducing Layer-wise Error Modeling Search (LEMS) and KFAC-SVD. LEMS advances rank allocation by introducing a layer-wise error surrogate that integrates local and global layer importance alongside a propagation bias, enabling effective global rank allocation via an ILP formulation. KFAC-SVD improves decomposition quality by utilizing token-wise statistics, mitigating the rank deficiency observed in prior Fisher-based SVD approaches. Across Mistral, Qwen3, and Llama3 model families, we show that LEMS consistently outperforms existing search strategies, delivering significant zero-shot accuracy gains of up to 4.8 p.p. that generalize to model sizes of 70B parameters, while KFAC-SVD achieves an average perplexity improvement of 15%. Project Page & Code: https://lems-svd.github.io} }
Endnote
%0 Conference Paper %T Advancing SVD-based LLM Compression via Layer-Wise Error Model Search %A Moritz Thoma %A Maximilian Groezinger %A Maximilian Forstenhäusler %A Emad Aghajanzadeh %A Manoj Rohit Vemparala %A Christos Anagnostopoulos %A Pierpaolo Mori %A Nael Fasfous %A Alexander Frickenstein %A Daniel Mueller-Gritschneder %A Ulf Schlichtmann %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-thoma26a %I PMLR %P 120620--120653 %U https://proceedings.mlr.press/v306/thoma26a.html %V 306 %X Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs. However, existing methods face two key limitations: (i) global rank allocation, where uncalibrated error proxies fail to capture complex error propagation, and (ii) decomposition quality, where Fisher-based estimators suffer from severe rank collapse. In this work, we address these limitations by introducing Layer-wise Error Modeling Search (LEMS) and KFAC-SVD. LEMS advances rank allocation by introducing a layer-wise error surrogate that integrates local and global layer importance alongside a propagation bias, enabling effective global rank allocation via an ILP formulation. KFAC-SVD improves decomposition quality by utilizing token-wise statistics, mitigating the rank deficiency observed in prior Fisher-based SVD approaches. Across Mistral, Qwen3, and Llama3 model families, we show that LEMS consistently outperforms existing search strategies, delivering significant zero-shot accuracy gains of up to 4.8 p.p. that generalize to model sizes of 70B parameters, while KFAC-SVD achieves an average perplexity improvement of 15%. Project Page & Code: https://lems-svd.github.io
APA
Thoma, M., Groezinger, M., Forstenhäusler, M., Aghajanzadeh, E., Vemparala, M.R., Anagnostopoulos, C., Mori, P., Fasfous, N., Frickenstein, A., Mueller-Gritschneder, D. & Schlichtmann, U.. (2026). Advancing SVD-based LLM Compression via Layer-Wise Error Model Search. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:120620-120653 Available from https://proceedings.mlr.press/v306/thoma26a.html.

Related Material