Implicit Regularization in Deep Tucker Factorization: Low-Rankness via Structured Sparsity

Kais Hariz, Hachem Kadri, Stéphane Ayache, Maher Moakher, Thierry Artières
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:2359-2367, 2024.

Abstract

We theoretically analyze the implicit regularization of deep learning for tensor completion. We show that deep Tucker factorization trained by gradient descent induces a structured sparse regularization. This leads to a characterization of the effect of the depth of the neural network on the implicit regularization and provides a potential explanation for the bias of gradient descent towards solutions with low multilinear rank. Numerical experiments confirm our theoretical findings and give insights into the behavior of gradient descent in deep tensor factorization.

Cite this Paper

BibTeX
@InProceedings{pmlr-v238-hariz24a, title = {Implicit Regularization in Deep {T}ucker Factorization: Low-Rankness via Structured Sparsity}, author = {Hariz, Kais and Kadri, Hachem and Ayache, St\'{e}phane and Moakher, Maher and Arti\`{e}res, Thierry}, booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics}, pages = {2359--2367}, year = {2024}, editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen}, volume = {238}, series = {Proceedings of Machine Learning Research}, month = {02--04 May}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v238/hariz24a/hariz24a.pdf}, url = {https://proceedings.mlr.press/v238/hariz24a.html}, abstract = {We theoretically analyze the implicit regularization of deep learning for tensor completion. We show that deep Tucker factorization trained by gradient descent induces a structured sparse regularization. This leads to a characterization of the effect of the depth of the neural network on the implicit regularization and provides a potential explanation for the bias of gradient descent towards solutions with low multilinear rank. Numerical experiments confirm our theoretical findings and give insights into the behavior of gradient descent in deep tensor factorization.} }
Endnote
%0 Conference Paper %T Implicit Regularization in Deep Tucker Factorization: Low-Rankness via Structured Sparsity %A Kais Hariz %A Hachem Kadri %A Stéphane Ayache %A Maher Moakher %A Thierry Artières %B Proceedings of The 27th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2024 %E Sanjoy Dasgupta %E Stephan Mandt %E Yingzhen Li %F pmlr-v238-hariz24a %I PMLR %P 2359--2367 %U https://proceedings.mlr.press/v238/hariz24a.html %V 238 %X We theoretically analyze the implicit regularization of deep learning for tensor completion. We show that deep Tucker factorization trained by gradient descent induces a structured sparse regularization. This leads to a characterization of the effect of the depth of the neural network on the implicit regularization and provides a potential explanation for the bias of gradient descent towards solutions with low multilinear rank. Numerical experiments confirm our theoretical findings and give insights into the behavior of gradient descent in deep tensor factorization.
APA
Hariz, K., Kadri, H., Ayache, S., Moakher, M. & Artières, T.. (2024). Implicit Regularization in Deep Tucker Factorization: Low-Rankness via Structured Sparsity. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 238:2359-2367 Available from https://proceedings.mlr.press/v238/hariz24a.html.

Related Material