Fisher-Rao Metric, Geometry, and Complexity of Neural Networks

Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, James Stokes
Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, PMLR 89:888-896, 2019.

Abstract

We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity — the Fisher-Rao norm — that possesses desirable invariance properties and is motivated by Information Geometry. We discover an analytical characterization of the new capacity measure, through which we establish norm-comparison inequalities and further show that the new measure serves as an umbrella for several existing norm-based complexity measures. We discuss upper bounds on the generalization error induced by the proposed measure. Extensive numerical experiments on CIFAR-10 support our theoretical findings. Our theoretical analysis rests on a key structural lemma about partial derivatives of multi-layer rectifier networks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v89-liang19a, title = {Fisher-Rao Metric, Geometry, and Complexity of Neural Networks}, author = {Liang, Tengyuan and Poggio, Tomaso and Rakhlin, Alexander and Stokes, James}, booktitle = {Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics}, pages = {888--896}, year = {2019}, editor = {Chaudhuri, Kamalika and Sugiyama, Masashi}, volume = {89}, series = {Proceedings of Machine Learning Research}, month = {16--18 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v89/liang19a/liang19a.pdf}, url = {https://proceedings.mlr.press/v89/liang19a.html}, abstract = {We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity — the Fisher-Rao norm — that possesses desirable invariance properties and is motivated by Information Geometry. We discover an analytical characterization of the new capacity measure, through which we establish norm-comparison inequalities and further show that the new measure serves as an umbrella for several existing norm-based complexity measures. We discuss upper bounds on the generalization error induced by the proposed measure. Extensive numerical experiments on CIFAR-10 support our theoretical findings. Our theoretical analysis rests on a key structural lemma about partial derivatives of multi-layer rectifier networks.} }
Endnote
%0 Conference Paper %T Fisher-Rao Metric, Geometry, and Complexity of Neural Networks %A Tengyuan Liang %A Tomaso Poggio %A Alexander Rakhlin %A James Stokes %B Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Masashi Sugiyama %F pmlr-v89-liang19a %I PMLR %P 888--896 %U https://proceedings.mlr.press/v89/liang19a.html %V 89 %X We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity — the Fisher-Rao norm — that possesses desirable invariance properties and is motivated by Information Geometry. We discover an analytical characterization of the new capacity measure, through which we establish norm-comparison inequalities and further show that the new measure serves as an umbrella for several existing norm-based complexity measures. We discuss upper bounds on the generalization error induced by the proposed measure. Extensive numerical experiments on CIFAR-10 support our theoretical findings. Our theoretical analysis rests on a key structural lemma about partial derivatives of multi-layer rectifier networks.
APA
Liang, T., Poggio, T., Rakhlin, A. & Stokes, J.. (2019). Fisher-Rao Metric, Geometry, and Complexity of Neural Networks. Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 89:888-896 Available from https://proceedings.mlr.press/v89/liang19a.html.

Related Material