The emergence of spectral universality in deep networks

Jeffrey Pennington, Samuel Schoenholz, Surya Ganguli
Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, PMLR 84:1924-1932, 2018.

Abstract

Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network’s input-output Jacobian around one at initialization can speed up learning by orders of magnitude. Therefore, to guide important design choices, it is important to build a full theoretical understanding of the spectra of Jacobians at initialization. To this end, we leverage powerful tools from free probability theory to provide a detailed analytic understanding of how a deep network’s Jacobian spectrum depends on various hyperparameters including the nonlinearity, the weight and bias distributions, and the depth. For a variety of nonlinearities, our work reveals the emergence of new universal limiting spectral distributions that remain concentrated around one even as the depth goes to infinity.

Cite this Paper

BibTeX
@InProceedings{pmlr-v84-pennington18a, title = {The emergence of spectral universality in deep networks}, author = {Pennington, Jeffrey and Schoenholz, Samuel and Ganguli, Surya}, booktitle = {Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics}, pages = {1924--1932}, year = {2018}, editor = {Storkey, Amos and Perez-Cruz, Fernando}, volume = {84}, series = {Proceedings of Machine Learning Research}, month = {09--11 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v84/pennington18a/pennington18a.pdf}, url = {https://proceedings.mlr.press/v84/pennington18a.html}, abstract = {Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network’s input-output Jacobian around one at initialization can speed up learning by orders of magnitude. Therefore, to guide important design choices, it is important to build a full theoretical understanding of the spectra of Jacobians at initialization. To this end, we leverage powerful tools from free probability theory to provide a detailed analytic understanding of how a deep network’s Jacobian spectrum depends on various hyperparameters including the nonlinearity, the weight and bias distributions, and the depth. For a variety of nonlinearities, our work reveals the emergence of new universal limiting spectral distributions that remain concentrated around one even as the depth goes to infinity.} }
Endnote
%0 Conference Paper %T The emergence of spectral universality in deep networks %A Jeffrey Pennington %A Samuel Schoenholz %A Surya Ganguli %B Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2018 %E Amos Storkey %E Fernando Perez-Cruz %F pmlr-v84-pennington18a %I PMLR %P 1924--1932 %U https://proceedings.mlr.press/v84/pennington18a.html %V 84 %X Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network’s input-output Jacobian around one at initialization can speed up learning by orders of magnitude. Therefore, to guide important design choices, it is important to build a full theoretical understanding of the spectra of Jacobians at initialization. To this end, we leverage powerful tools from free probability theory to provide a detailed analytic understanding of how a deep network’s Jacobian spectrum depends on various hyperparameters including the nonlinearity, the weight and bias distributions, and the depth. For a variety of nonlinearities, our work reveals the emergence of new universal limiting spectral distributions that remain concentrated around one even as the depth goes to infinity.
APA
Pennington, J., Schoenholz, S. & Ganguli, S.. (2018). The emergence of spectral universality in deep networks. Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 84:1924-1932 Available from https://proceedings.mlr.press/v84/pennington18a.html.

Related Material