Essentially No Barriers in Neural Network Energy Landscape

Felix Draxler, Kambis Veschgini, Manfred Salmhofer, Fred Hamprecht
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:1309-1318, 2018.

Abstract

Training neural networks involves finding minima of a high-dimensional non-convex loss function. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that minima are perhaps best seen as points on a single connected manifold of low loss, rather than as the bottoms of distinct valleys.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-draxler18a, title = {Essentially No Barriers in Neural Network Energy Landscape}, author = {Draxler, Felix and Veschgini, Kambis and Salmhofer, Manfred and Hamprecht, Fred}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {1309--1318}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/draxler18a/draxler18a.pdf}, url = {https://proceedings.mlr.press/v80/draxler18a.html}, abstract = {Training neural networks involves finding minima of a high-dimensional non-convex loss function. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that minima are perhaps best seen as points on a single connected manifold of low loss, rather than as the bottoms of distinct valleys.} }
Endnote
%0 Conference Paper %T Essentially No Barriers in Neural Network Energy Landscape %A Felix Draxler %A Kambis Veschgini %A Manfred Salmhofer %A Fred Hamprecht %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-draxler18a %I PMLR %P 1309--1318 %U https://proceedings.mlr.press/v80/draxler18a.html %V 80 %X Training neural networks involves finding minima of a high-dimensional non-convex loss function. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that minima are perhaps best seen as points on a single connected manifold of low loss, rather than as the bottoms of distinct valleys.
APA
Draxler, F., Veschgini, K., Salmhofer, M. & Hamprecht, F.. (2018). Essentially No Barriers in Neural Network Energy Landscape. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:1309-1318 Available from https://proceedings.mlr.press/v80/draxler18a.html.

Related Material