Being a Bit Frequentist Improves Bayesian Neural Networks

Agustinus Kristiadi, Matthias Hein, Philipp Hennig
Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR 151:529-545, 2022.

Abstract

Despite their compelling theoretical properties, Bayesian neural networks (BNNs) tend to perform worse than frequentist methods in classification-based uncertainty quantification (UQ) tasks such as out-of-distribution (OOD) detection. In this paper, based on empirical findings in prior works, we hypothesize that this issue is because even recent Bayesian methods have never considered OOD data in their training processes, even though this “OOD training” technique is an integral part of state-of-the-art frequentist UQ methods. To validate this, we treat OOD data as a first-class citizen in BNN training by exploring four different ways of incorporating OOD data into Bayesian inference. We show in extensive experiments that OOD-trained BNNs are competitive to recent frequentist baselines. This work thus provides strong baselines for future work in Bayesian UQ.

Cite this Paper

BibTeX
@InProceedings{pmlr-v151-kristiadi22a, title = { Being a Bit Frequentist Improves Bayesian Neural Networks }, author = {Kristiadi, Agustinus and Hein, Matthias and Hennig, Philipp}, booktitle = {Proceedings of The 25th International Conference on Artificial Intelligence and Statistics}, pages = {529--545}, year = {2022}, editor = {Camps-Valls, Gustau and Ruiz, Francisco J. R. and Valera, Isabel}, volume = {151}, series = {Proceedings of Machine Learning Research}, month = {28--30 Mar}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v151/kristiadi22a/kristiadi22a.pdf}, url = {https://proceedings.mlr.press/v151/kristiadi22a.html}, abstract = { Despite their compelling theoretical properties, Bayesian neural networks (BNNs) tend to perform worse than frequentist methods in classification-based uncertainty quantification (UQ) tasks such as out-of-distribution (OOD) detection. In this paper, based on empirical findings in prior works, we hypothesize that this issue is because even recent Bayesian methods have never considered OOD data in their training processes, even though this “OOD training” technique is an integral part of state-of-the-art frequentist UQ methods. To validate this, we treat OOD data as a first-class citizen in BNN training by exploring four different ways of incorporating OOD data into Bayesian inference. We show in extensive experiments that OOD-trained BNNs are competitive to recent frequentist baselines. This work thus provides strong baselines for future work in Bayesian UQ. } }
Endnote
%0 Conference Paper %T Being a Bit Frequentist Improves Bayesian Neural Networks %A Agustinus Kristiadi %A Matthias Hein %A Philipp Hennig %B Proceedings of The 25th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2022 %E Gustau Camps-Valls %E Francisco J. R. Ruiz %E Isabel Valera %F pmlr-v151-kristiadi22a %I PMLR %P 529--545 %U https://proceedings.mlr.press/v151/kristiadi22a.html %V 151 %X Despite their compelling theoretical properties, Bayesian neural networks (BNNs) tend to perform worse than frequentist methods in classification-based uncertainty quantification (UQ) tasks such as out-of-distribution (OOD) detection. In this paper, based on empirical findings in prior works, we hypothesize that this issue is because even recent Bayesian methods have never considered OOD data in their training processes, even though this “OOD training” technique is an integral part of state-of-the-art frequentist UQ methods. To validate this, we treat OOD data as a first-class citizen in BNN training by exploring four different ways of incorporating OOD data into Bayesian inference. We show in extensive experiments that OOD-trained BNNs are competitive to recent frequentist baselines. This work thus provides strong baselines for future work in Bayesian UQ.
APA
Kristiadi, A., Hein, M. & Hennig, P.. (2022). Being a Bit Frequentist Improves Bayesian Neural Networks . Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 151:529-545 Available from https://proceedings.mlr.press/v151/kristiadi22a.html.

Related Material