Asymptotics of Ridge Regression in Convolutional Models

Mojtaba Sahraee-Ardakan, Tung Mai, Anup Rao, Ryan A. Rossi, Sundeep Rangan, Alyson K Fletcher
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:9265-9275, 2021.

Abstract

Understanding generalization and estimation error of estimators for simple models such as linear and generalized linear models has attracted a lot of attention recently. This is in part due to an interesting observation made in machine learning community that highly over-parameterized neural networks achieve zero training error, and yet they are able to generalize well over the test samples. This phenomenon is captured by the so called double descent curve, where the generalization error starts decreasing again after the interpolation threshold. A series of recent works tried to explain such phenomenon for simple models. In this work, we analyze the asymptotics of estimation error in ridge estimators for convolutional linear models. These convolutional inverse problems, also known as deconvolution, naturally arise in different fields such as seismology, imaging, and acoustics among others. Our results hold for a large class of input distributions that include i.i.d. features as a special case. We derive exact formulae for estimation error of ridge estimators that hold in a certain high-dimensional regime. We show the double descent phenomenon in our experiments for convolutional models and show that our theoretical results match the experiments.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-sahraee-ardakan21a, title = {Asymptotics of Ridge Regression in Convolutional Models}, author = {Sahraee-Ardakan, Mojtaba and Mai, Tung and Rao, Anup and Rossi, Ryan A. and Rangan, Sundeep and Fletcher, Alyson K}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {9265--9275}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/sahraee-ardakan21a/sahraee-ardakan21a.pdf}, url = {https://proceedings.mlr.press/v139/sahraee-ardakan21a.html}, abstract = {Understanding generalization and estimation error of estimators for simple models such as linear and generalized linear models has attracted a lot of attention recently. This is in part due to an interesting observation made in machine learning community that highly over-parameterized neural networks achieve zero training error, and yet they are able to generalize well over the test samples. This phenomenon is captured by the so called double descent curve, where the generalization error starts decreasing again after the interpolation threshold. A series of recent works tried to explain such phenomenon for simple models. In this work, we analyze the asymptotics of estimation error in ridge estimators for convolutional linear models. These convolutional inverse problems, also known as deconvolution, naturally arise in different fields such as seismology, imaging, and acoustics among others. Our results hold for a large class of input distributions that include i.i.d. features as a special case. We derive exact formulae for estimation error of ridge estimators that hold in a certain high-dimensional regime. We show the double descent phenomenon in our experiments for convolutional models and show that our theoretical results match the experiments.} }
Endnote
%0 Conference Paper %T Asymptotics of Ridge Regression in Convolutional Models %A Mojtaba Sahraee-Ardakan %A Tung Mai %A Anup Rao %A Ryan A. Rossi %A Sundeep Rangan %A Alyson K Fletcher %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-sahraee-ardakan21a %I PMLR %P 9265--9275 %U https://proceedings.mlr.press/v139/sahraee-ardakan21a.html %V 139 %X Understanding generalization and estimation error of estimators for simple models such as linear and generalized linear models has attracted a lot of attention recently. This is in part due to an interesting observation made in machine learning community that highly over-parameterized neural networks achieve zero training error, and yet they are able to generalize well over the test samples. This phenomenon is captured by the so called double descent curve, where the generalization error starts decreasing again after the interpolation threshold. A series of recent works tried to explain such phenomenon for simple models. In this work, we analyze the asymptotics of estimation error in ridge estimators for convolutional linear models. These convolutional inverse problems, also known as deconvolution, naturally arise in different fields such as seismology, imaging, and acoustics among others. Our results hold for a large class of input distributions that include i.i.d. features as a special case. We derive exact formulae for estimation error of ridge estimators that hold in a certain high-dimensional regime. We show the double descent phenomenon in our experiments for convolutional models and show that our theoretical results match the experiments.
APA
Sahraee-Ardakan, M., Mai, T., Rao, A., Rossi, R.A., Rangan, S. & Fletcher, A.K.. (2021). Asymptotics of Ridge Regression in Convolutional Models. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:9265-9275 Available from https://proceedings.mlr.press/v139/sahraee-ardakan21a.html.

Related Material