Efficient Low-Dimensional Compression of Overparameterized Models

Soo Min Kwon, Zekai Zhang, Dogyoon Song, Laura Balzano, Qing Qu
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:1009-1017, 2024.

Abstract

In this work, we present a novel approach for compressing overparameterized models, developed through studying their learning dynamics. We observe that for many deep models, updates to the weight matrices occur within a low-dimensional invariant subspace. For deep linear models, we demonstrate that their principal components are fitted incrementally within a small subspace, and use these insights to propose a compression algorithm for deep linear networks that involve decreasing the width of their intermediate layers. We empirically evaluate the effectiveness of our compression technique on matrix recovery problems. Remarkably, by using an initialization that exploits the structure of the problem, we observe that our compressed network converges faster than the original network, consistently yielding smaller recovery errors. We substantiate this observation by developing a theory focused on deep matrix factorization. Finally, we empirically demonstrate how our compressed model has the potential to improve the utility of deep nonlinear models. Overall, our algorithm improves the training efficiency by more than 2x, without compromising generalization.

Cite this Paper

BibTeX
@InProceedings{pmlr-v238-min-kwon24a, title = {Efficient Low-Dimensional Compression of Overparameterized Models}, author = {Min Kwon, Soo and Zhang, Zekai and Song, Dogyoon and Balzano, Laura and Qu, Qing}, booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics}, pages = {1009--1017}, year = {2024}, editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen}, volume = {238}, series = {Proceedings of Machine Learning Research}, month = {02--04 May}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v238/min-kwon24a/min-kwon24a.pdf}, url = {https://proceedings.mlr.press/v238/min-kwon24a.html}, abstract = {In this work, we present a novel approach for compressing overparameterized models, developed through studying their learning dynamics. We observe that for many deep models, updates to the weight matrices occur within a low-dimensional invariant subspace. For deep linear models, we demonstrate that their principal components are fitted incrementally within a small subspace, and use these insights to propose a compression algorithm for deep linear networks that involve decreasing the width of their intermediate layers. We empirically evaluate the effectiveness of our compression technique on matrix recovery problems. Remarkably, by using an initialization that exploits the structure of the problem, we observe that our compressed network converges faster than the original network, consistently yielding smaller recovery errors. We substantiate this observation by developing a theory focused on deep matrix factorization. Finally, we empirically demonstrate how our compressed model has the potential to improve the utility of deep nonlinear models. Overall, our algorithm improves the training efficiency by more than 2x, without compromising generalization.} }
Endnote
%0 Conference Paper %T Efficient Low-Dimensional Compression of Overparameterized Models %A Soo Min Kwon %A Zekai Zhang %A Dogyoon Song %A Laura Balzano %A Qing Qu %B Proceedings of The 27th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2024 %E Sanjoy Dasgupta %E Stephan Mandt %E Yingzhen Li %F pmlr-v238-min-kwon24a %I PMLR %P 1009--1017 %U https://proceedings.mlr.press/v238/min-kwon24a.html %V 238 %X In this work, we present a novel approach for compressing overparameterized models, developed through studying their learning dynamics. We observe that for many deep models, updates to the weight matrices occur within a low-dimensional invariant subspace. For deep linear models, we demonstrate that their principal components are fitted incrementally within a small subspace, and use these insights to propose a compression algorithm for deep linear networks that involve decreasing the width of their intermediate layers. We empirically evaluate the effectiveness of our compression technique on matrix recovery problems. Remarkably, by using an initialization that exploits the structure of the problem, we observe that our compressed network converges faster than the original network, consistently yielding smaller recovery errors. We substantiate this observation by developing a theory focused on deep matrix factorization. Finally, we empirically demonstrate how our compressed model has the potential to improve the utility of deep nonlinear models. Overall, our algorithm improves the training efficiency by more than 2x, without compromising generalization.
APA
Min Kwon, S., Zhang, Z., Song, D., Balzano, L. & Qu, Q.. (2024). Efficient Low-Dimensional Compression of Overparameterized Models. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 238:1009-1017 Available from https://proceedings.mlr.press/v238/min-kwon24a.html.

Related Material