On the Expressive Power of Deep Neural Networks

Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl-Dickstein
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:2847-2854, 2017.

Abstract

We propose a new approach to the problem of neural network expressivity, which seeks to characterize how structural properties of a neural network family affect the functions it is able to compute. Our approach is based on an interrelated set of measures of expressivity, unified by the novel notion of trajectory length, which measures how the output of a network changes as the input sweeps along a one-dimensional path. Our findings show that: (1) The complexity of the computed function grows exponentially with depth (2) All weights are not equal: trained networks are more sensitive to their lower (initial) layer weights (3) Trajectory regularization is a simpler alternative to batch normalization, with the same performance.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-raghu17a, title = {On the Expressive Power of Deep Neural Networks}, author = {Maithra Raghu and Ben Poole and Jon Kleinberg and Surya Ganguli and Jascha Sohl-Dickstein}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {2847--2854}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/raghu17a/raghu17a.pdf}, url = {https://proceedings.mlr.press/v70/raghu17a.html}, abstract = {We propose a new approach to the problem of neural network expressivity, which seeks to characterize how structural properties of a neural network family affect the functions it is able to compute. Our approach is based on an interrelated set of measures of expressivity, unified by the novel notion of trajectory length, which measures how the output of a network changes as the input sweeps along a one-dimensional path. Our findings show that: (1) The complexity of the computed function grows exponentially with depth (2) All weights are not equal: trained networks are more sensitive to their lower (initial) layer weights (3) Trajectory regularization is a simpler alternative to batch normalization, with the same performance.} }
Endnote
%0 Conference Paper %T On the Expressive Power of Deep Neural Networks %A Maithra Raghu %A Ben Poole %A Jon Kleinberg %A Surya Ganguli %A Jascha Sohl-Dickstein %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-raghu17a %I PMLR %P 2847--2854 %U https://proceedings.mlr.press/v70/raghu17a.html %V 70 %X We propose a new approach to the problem of neural network expressivity, which seeks to characterize how structural properties of a neural network family affect the functions it is able to compute. Our approach is based on an interrelated set of measures of expressivity, unified by the novel notion of trajectory length, which measures how the output of a network changes as the input sweeps along a one-dimensional path. Our findings show that: (1) The complexity of the computed function grows exponentially with depth (2) All weights are not equal: trained networks are more sensitive to their lower (initial) layer weights (3) Trajectory regularization is a simpler alternative to batch normalization, with the same performance.
APA
Raghu, M., Poole, B., Kleinberg, J., Ganguli, S. & Sohl-Dickstein, J.. (2017). On the Expressive Power of Deep Neural Networks. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:2847-2854 Available from https://proceedings.mlr.press/v70/raghu17a.html.

Related Material