Paths and Ambient Spaces in Neural Loss Landscapes

Daniel Dold, Julius Kobialka, Nicolai Palm, Emanuel Sommer, David Rügamer, Oliver Dürr
Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:10-18, 2025.

Abstract

Understanding the structure of neural network loss surfaces, particularly the emergence of low-loss tunnels, is critical for advancing neural network theory and practice. In this paper, we propose a novel approach to directly embed loss tunnels into the loss landscape of neural networks. Exploring the properties of these loss tunnels offers new insights into their length and structure and sheds light on some common misconceptions. We then apply our approach to Bayesian neural networks, where we improve subspace inference by identifying pitfalls and proposing a more natural prior that better guides the sampling procedure.

Cite this Paper

BibTeX
@InProceedings{pmlr-v258-dold25a, title = {Paths and Ambient Spaces in Neural Loss Landscapes}, author = {Dold, Daniel and Kobialka, Julius and Palm, Nicolai and Sommer, Emanuel and R{\"u}gamer, David and D{\"u}rr, Oliver}, booktitle = {Proceedings of The 28th International Conference on Artificial Intelligence and Statistics}, pages = {10--18}, year = {2025}, editor = {Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz}, volume = {258}, series = {Proceedings of Machine Learning Research}, month = {03--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v258/main/assets/dold25a/dold25a.pdf}, url = {https://proceedings.mlr.press/v258/dold25a.html}, abstract = {Understanding the structure of neural network loss surfaces, particularly the emergence of low-loss tunnels, is critical for advancing neural network theory and practice. In this paper, we propose a novel approach to directly embed loss tunnels into the loss landscape of neural networks. Exploring the properties of these loss tunnels offers new insights into their length and structure and sheds light on some common misconceptions. We then apply our approach to Bayesian neural networks, where we improve subspace inference by identifying pitfalls and proposing a more natural prior that better guides the sampling procedure.} }
Endnote
%0 Conference Paper %T Paths and Ambient Spaces in Neural Loss Landscapes %A Daniel Dold %A Julius Kobialka %A Nicolai Palm %A Emanuel Sommer %A David Rügamer %A Oliver Dürr %B Proceedings of The 28th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2025 %E Yingzhen Li %E Stephan Mandt %E Shipra Agrawal %E Emtiyaz Khan %F pmlr-v258-dold25a %I PMLR %P 10--18 %U https://proceedings.mlr.press/v258/dold25a.html %V 258 %X Understanding the structure of neural network loss surfaces, particularly the emergence of low-loss tunnels, is critical for advancing neural network theory and practice. In this paper, we propose a novel approach to directly embed loss tunnels into the loss landscape of neural networks. Exploring the properties of these loss tunnels offers new insights into their length and structure and sheds light on some common misconceptions. We then apply our approach to Bayesian neural networks, where we improve subspace inference by identifying pitfalls and proposing a more natural prior that better guides the sampling procedure.
APA
Dold, D., Kobialka, J., Palm, N., Sommer, E., Rügamer, D. & Dürr, O.. (2025). Paths and Ambient Spaces in Neural Loss Landscapes. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 258:10-18 Available from https://proceedings.mlr.press/v258/dold25a.html.

Related Material