On Connected Sublevel Sets in Deep Learning

Quynh Nguyen
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:4790-4799, 2019.

Abstract

This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large global valley.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-nguyen19a, title = {On Connected Sublevel Sets in Deep Learning}, author = {Nguyen, Quynh}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {4790--4799}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/nguyen19a/nguyen19a.pdf}, url = {https://proceedings.mlr.press/v97/nguyen19a.html}, abstract = {This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large global valley.} }
Endnote
%0 Conference Paper %T On Connected Sublevel Sets in Deep Learning %A Quynh Nguyen %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-nguyen19a %I PMLR %P 4790--4799 %U https://proceedings.mlr.press/v97/nguyen19a.html %V 97 %X This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large global valley.
APA
Nguyen, Q.. (2019). On Connected Sublevel Sets in Deep Learning. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:4790-4799 Available from https://proceedings.mlr.press/v97/nguyen19a.html.

Related Material