Connectedness of loss landscapes via the lens of Morse theory

Danil Akhtiamov, Matt Thomson
Proceedings of the 1st NeurIPS Workshop on Symmetry and Geometry in Neural Representations, PMLR 197:171-181, 2023.

Abstract

Mode connectivity is a recently discovered property of neural networks stating that two weight configurations of small loss can usually be connected by a path of small loss. The mode connectivity property is interesting practically as it has applications to design of optimizers with better generalization properties and various other applied topics as well as theoretically as it suggests that loss landscapes of deep networks have very nice properties even though they are known to be highly non-convex. The goal of this work is to study connectedness of loss landscapes via the lens of Morse theory. A brief introduction to Morse theory is provided.

Cite this Paper

BibTeX
@InProceedings{pmlr-v197-akhtiamov23a, title = {Connectedness of loss landscapes via the lens of Morse theory}, author = {Akhtiamov, Danil and Thomson, Matt}, booktitle = {Proceedings of the 1st NeurIPS Workshop on Symmetry and Geometry in Neural Representations}, pages = {171--181}, year = {2023}, editor = {Sanborn, Sophia and Shewmake, Christian and Azeglio, Simone and Di Bernardo, Arianna and Miolane, Nina}, volume = {197}, series = {Proceedings of Machine Learning Research}, month = {03 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v197/akhtiamov23a/akhtiamov23a.pdf}, url = {https://proceedings.mlr.press/v197/akhtiamov23a.html}, abstract = {Mode connectivity is a recently discovered property of neural networks stating that two weight configurations of small loss can usually be connected by a path of small loss. The mode connectivity property is interesting practically as it has applications to design of optimizers with better generalization properties and various other applied topics as well as theoretically as it suggests that loss landscapes of deep networks have very nice properties even though they are known to be highly non-convex. The goal of this work is to study connectedness of loss landscapes via the lens of Morse theory. A brief introduction to Morse theory is provided.} }
Endnote
%0 Conference Paper %T Connectedness of loss landscapes via the lens of Morse theory %A Danil Akhtiamov %A Matt Thomson %B Proceedings of the 1st NeurIPS Workshop on Symmetry and Geometry in Neural Representations %C Proceedings of Machine Learning Research %D 2023 %E Sophia Sanborn %E Christian Shewmake %E Simone Azeglio %E Arianna Di Bernardo %E Nina Miolane %F pmlr-v197-akhtiamov23a %I PMLR %P 171--181 %U https://proceedings.mlr.press/v197/akhtiamov23a.html %V 197 %X Mode connectivity is a recently discovered property of neural networks stating that two weight configurations of small loss can usually be connected by a path of small loss. The mode connectivity property is interesting practically as it has applications to design of optimizers with better generalization properties and various other applied topics as well as theoretically as it suggests that loss landscapes of deep networks have very nice properties even though they are known to be highly non-convex. The goal of this work is to study connectedness of loss landscapes via the lens of Morse theory. A brief introduction to Morse theory is provided.
APA
Akhtiamov, D. & Thomson, M.. (2023). Connectedness of loss landscapes via the lens of Morse theory. Proceedings of the 1st NeurIPS Workshop on Symmetry and Geometry in Neural Representations, in Proceedings of Machine Learning Research 197:171-181 Available from https://proceedings.mlr.press/v197/akhtiamov23a.html.

Related Material