Connecting the Dots: Is Mode-Connectedness the Key to Feasible Sample-Based Inference in Bayesian Neural Networks?

Emanuel Sommer, Lisa Wimmer, Theodore Papamarkou, Ludwig Bothmann, Bernd Bischl, David Rügamer
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:45988-46018, 2024.

Abstract

A major challenge in sample-based inference (SBI) for Bayesian neural networks is the size and structure of the networks’ parameter space. Our work shows that successful SBI is possible by embracing the characteristic relationship between weight and function space, uncovering a systematic link between overparameterization and the difficulty of the sampling problem. Through extensive experiments, we establish practical guidelines for sampling and convergence diagnosis. As a result, we present a deep ensemble initialized approach as an effective solution with competitive performance and uncertainty quantification.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-sommer24a, title = {Connecting the Dots: Is Mode-Connectedness the Key to Feasible Sample-Based Inference in {B}ayesian Neural Networks?}, author = {Sommer, Emanuel and Wimmer, Lisa and Papamarkou, Theodore and Bothmann, Ludwig and Bischl, Bernd and R\"{u}gamer, David}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {45988--46018}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/sommer24a/sommer24a.pdf}, url = {https://proceedings.mlr.press/v235/sommer24a.html}, abstract = {A major challenge in sample-based inference (SBI) for Bayesian neural networks is the size and structure of the networks’ parameter space. Our work shows that successful SBI is possible by embracing the characteristic relationship between weight and function space, uncovering a systematic link between overparameterization and the difficulty of the sampling problem. Through extensive experiments, we establish practical guidelines for sampling and convergence diagnosis. As a result, we present a deep ensemble initialized approach as an effective solution with competitive performance and uncertainty quantification.} }
Endnote
%0 Conference Paper %T Connecting the Dots: Is Mode-Connectedness the Key to Feasible Sample-Based Inference in Bayesian Neural Networks? %A Emanuel Sommer %A Lisa Wimmer %A Theodore Papamarkou %A Ludwig Bothmann %A Bernd Bischl %A David Rügamer %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-sommer24a %I PMLR %P 45988--46018 %U https://proceedings.mlr.press/v235/sommer24a.html %V 235 %X A major challenge in sample-based inference (SBI) for Bayesian neural networks is the size and structure of the networks’ parameter space. Our work shows that successful SBI is possible by embracing the characteristic relationship between weight and function space, uncovering a systematic link between overparameterization and the difficulty of the sampling problem. Through extensive experiments, we establish practical guidelines for sampling and convergence diagnosis. As a result, we present a deep ensemble initialized approach as an effective solution with competitive performance and uncertainty quantification.
APA
Sommer, E., Wimmer, L., Papamarkou, T., Bothmann, L., Bischl, B. & Rügamer, D.. (2024). Connecting the Dots: Is Mode-Connectedness the Key to Feasible Sample-Based Inference in Bayesian Neural Networks?. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:45988-46018 Available from https://proceedings.mlr.press/v235/sommer24a.html.

Related Material