Tropical Geometry of Deep Neural Networks

Liwen Zhang, Gregory Naitzat, Lek-Heng Lim
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:5824-5832, 2018.

Abstract

We establish, for the first time, explicit connections between feedforward neural networks with ReLU activation and tropical geometry — we show that the family of such neural networks is equivalent to the family of tropical rational maps. Among other things, we deduce that feedforward ReLU neural networks with one hidden layer can be characterized by zonotopes, which serve as building blocks for deeper networks; we relate decision boundaries of such neural networks to tropical hypersurfaces, a major object of study in tropical geometry; and we prove that linear regions of such neural networks correspond to vertices of polytopes associated with tropical rational functions. An insight from our tropical formulation is that a deeper network is exponentially more expressive than a shallow network.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-zhang18i, title = {Tropical Geometry of Deep Neural Networks}, author = {Zhang, Liwen and Naitzat, Gregory and Lim, Lek-Heng}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {5824--5832}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/zhang18i/zhang18i.pdf}, url = {https://proceedings.mlr.press/v80/zhang18i.html}, abstract = {We establish, for the first time, explicit connections between feedforward neural networks with ReLU activation and tropical geometry — we show that the family of such neural networks is equivalent to the family of tropical rational maps. Among other things, we deduce that feedforward ReLU neural networks with one hidden layer can be characterized by zonotopes, which serve as building blocks for deeper networks; we relate decision boundaries of such neural networks to tropical hypersurfaces, a major object of study in tropical geometry; and we prove that linear regions of such neural networks correspond to vertices of polytopes associated with tropical rational functions. An insight from our tropical formulation is that a deeper network is exponentially more expressive than a shallow network.} }
Endnote
%0 Conference Paper %T Tropical Geometry of Deep Neural Networks %A Liwen Zhang %A Gregory Naitzat %A Lek-Heng Lim %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-zhang18i %I PMLR %P 5824--5832 %U https://proceedings.mlr.press/v80/zhang18i.html %V 80 %X We establish, for the first time, explicit connections between feedforward neural networks with ReLU activation and tropical geometry — we show that the family of such neural networks is equivalent to the family of tropical rational maps. Among other things, we deduce that feedforward ReLU neural networks with one hidden layer can be characterized by zonotopes, which serve as building blocks for deeper networks; we relate decision boundaries of such neural networks to tropical hypersurfaces, a major object of study in tropical geometry; and we prove that linear regions of such neural networks correspond to vertices of polytopes associated with tropical rational functions. An insight from our tropical formulation is that a deeper network is exponentially more expressive than a shallow network.
APA
Zhang, L., Naitzat, G. & Lim, L.. (2018). Tropical Geometry of Deep Neural Networks. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:5824-5832 Available from https://proceedings.mlr.press/v80/zhang18i.html.

Related Material