Implicit Regularization with Polynomial Growth in Deep Tensor Factorization

Kais Hariz, Hachem Kadri, Stephane Ayache, Maher Moakher, Thierry Artieres
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:8484-8501, 2022.

Abstract

We study the implicit regularization effects of deep learning in tensor factorization. While implicit regularization in deep matrix and ’shallow’ tensor factorization via linear and certain type of non-linear neural networks promotes low-rank solutions with at most quadratic growth, we show that its effect in deep tensor factorization grows polynomially with the depth of the network. This provides a remarkably faithful description of the observed experimental behaviour. Using numerical experiments, we demonstrate the benefits of this implicit regularization in yielding a more accurate estimation and better convergence properties.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-hariz22a, title = {Implicit Regularization with Polynomial Growth in Deep Tensor Factorization}, author = {Hariz, Kais and Kadri, Hachem and Ayache, Stephane and Moakher, Maher and Artieres, Thierry}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {8484--8501}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/hariz22a/hariz22a.pdf}, url = {https://proceedings.mlr.press/v162/hariz22a.html}, abstract = {We study the implicit regularization effects of deep learning in tensor factorization. While implicit regularization in deep matrix and ’shallow’ tensor factorization via linear and certain type of non-linear neural networks promotes low-rank solutions with at most quadratic growth, we show that its effect in deep tensor factorization grows polynomially with the depth of the network. This provides a remarkably faithful description of the observed experimental behaviour. Using numerical experiments, we demonstrate the benefits of this implicit regularization in yielding a more accurate estimation and better convergence properties.} }
Endnote
%0 Conference Paper %T Implicit Regularization with Polynomial Growth in Deep Tensor Factorization %A Kais Hariz %A Hachem Kadri %A Stephane Ayache %A Maher Moakher %A Thierry Artieres %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-hariz22a %I PMLR %P 8484--8501 %U https://proceedings.mlr.press/v162/hariz22a.html %V 162 %X We study the implicit regularization effects of deep learning in tensor factorization. While implicit regularization in deep matrix and ’shallow’ tensor factorization via linear and certain type of non-linear neural networks promotes low-rank solutions with at most quadratic growth, we show that its effect in deep tensor factorization grows polynomially with the depth of the network. This provides a remarkably faithful description of the observed experimental behaviour. Using numerical experiments, we demonstrate the benefits of this implicit regularization in yielding a more accurate estimation and better convergence properties.
APA
Hariz, K., Kadri, H., Ayache, S., Moakher, M. & Artieres, T.. (2022). Implicit Regularization with Polynomial Growth in Deep Tensor Factorization. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:8484-8501 Available from https://proceedings.mlr.press/v162/hariz22a.html.

Related Material