Neural Tangent Kernel Analysis of Deep Narrow Neural Networks

Jongmin Lee, Joo Young Choi, Ernest K Ryu, Albert No
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:12282-12351, 2022.

Abstract

The tremendous recent progress in analyzing the training dynamics of overparameterized neural networks has primarily focused on wide networks and therefore does not sufficiently address the role of depth in deep learning. In this work, we present the first trainability guarantee of infinitely deep but narrow neural networks. We study the infinite-depth limit of a multilayer perceptron (MLP) with a specific initialization and establish a trainability guarantee using the NTK theory. We then extend the analysis to an infinitely deep convolutional neural network (CNN) and perform brief experiments.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-lee22a, title = {Neural Tangent Kernel Analysis of Deep Narrow Neural Networks}, author = {Lee, Jongmin and Choi, Joo Young and Ryu, Ernest K and No, Albert}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {12282--12351}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/lee22a/lee22a.pdf}, url = {https://proceedings.mlr.press/v162/lee22a.html}, abstract = {The tremendous recent progress in analyzing the training dynamics of overparameterized neural networks has primarily focused on wide networks and therefore does not sufficiently address the role of depth in deep learning. In this work, we present the first trainability guarantee of infinitely deep but narrow neural networks. We study the infinite-depth limit of a multilayer perceptron (MLP) with a specific initialization and establish a trainability guarantee using the NTK theory. We then extend the analysis to an infinitely deep convolutional neural network (CNN) and perform brief experiments.} }
Endnote
%0 Conference Paper %T Neural Tangent Kernel Analysis of Deep Narrow Neural Networks %A Jongmin Lee %A Joo Young Choi %A Ernest K Ryu %A Albert No %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-lee22a %I PMLR %P 12282--12351 %U https://proceedings.mlr.press/v162/lee22a.html %V 162 %X The tremendous recent progress in analyzing the training dynamics of overparameterized neural networks has primarily focused on wide networks and therefore does not sufficiently address the role of depth in deep learning. In this work, we present the first trainability guarantee of infinitely deep but narrow neural networks. We study the infinite-depth limit of a multilayer perceptron (MLP) with a specific initialization and establish a trainability guarantee using the NTK theory. We then extend the analysis to an infinitely deep convolutional neural network (CNN) and perform brief experiments.
APA
Lee, J., Choi, J.Y., Ryu, E.K. & No, A.. (2022). Neural Tangent Kernel Analysis of Deep Narrow Neural Networks. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:12282-12351 Available from https://proceedings.mlr.press/v162/lee22a.html.

Related Material