Spurious Valleys and Clustering Behavior of Neural Networks

Samuele Pollaci
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:28079-28099, 2023.

Abstract

Neural networks constitute a class of functions that are typically non-surjective, with high-dimensional fibers and complicated image. We prove two main results concerning the geometry of the loss landscape of a neural network. First, we provide an explicit effective bound on the sizes of the hidden layers so that the loss landscape has no spurious valleys, which guarantees the success of gradient descent methods. Second, we present a novel method for analyzing whether a given neural network architecture with monomial activation function can represent a target function of interest. The core of our analysis method is the study of a specific set of error values, and its behavior depending on different training datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-pollaci23a, title = {Spurious Valleys and Clustering Behavior of Neural Networks}, author = {Pollaci, Samuele}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {28079--28099}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/pollaci23a/pollaci23a.pdf}, url = {https://proceedings.mlr.press/v202/pollaci23a.html}, abstract = {Neural networks constitute a class of functions that are typically non-surjective, with high-dimensional fibers and complicated image. We prove two main results concerning the geometry of the loss landscape of a neural network. First, we provide an explicit effective bound on the sizes of the hidden layers so that the loss landscape has no spurious valleys, which guarantees the success of gradient descent methods. Second, we present a novel method for analyzing whether a given neural network architecture with monomial activation function can represent a target function of interest. The core of our analysis method is the study of a specific set of error values, and its behavior depending on different training datasets.} }
Endnote
%0 Conference Paper %T Spurious Valleys and Clustering Behavior of Neural Networks %A Samuele Pollaci %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-pollaci23a %I PMLR %P 28079--28099 %U https://proceedings.mlr.press/v202/pollaci23a.html %V 202 %X Neural networks constitute a class of functions that are typically non-surjective, with high-dimensional fibers and complicated image. We prove two main results concerning the geometry of the loss landscape of a neural network. First, we provide an explicit effective bound on the sizes of the hidden layers so that the loss landscape has no spurious valleys, which guarantees the success of gradient descent methods. Second, we present a novel method for analyzing whether a given neural network architecture with monomial activation function can represent a target function of interest. The core of our analysis method is the study of a specific set of error values, and its behavior depending on different training datasets.
APA
Pollaci, S.. (2023). Spurious Valleys and Clustering Behavior of Neural Networks. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:28079-28099 Available from https://proceedings.mlr.press/v202/pollaci23a.html.

Related Material