Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning

Sebastian Farquhar, Michael A. Osborne, Yarin Gal
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:1352-1362, 2020.

Abstract

We propose Radial Bayesian Neural Networks (BNNs): a variational approximate posterior for BNNs which scales well to large models. Unlike scalable Bayesian deep learning methods like deep ensembles that have discrete support (assign exactly zero probability almost everywhere in weight-space) Radial BNNs maintain full support; letting them act as a prior for continual learning and avoiding the a priori implausibility of discrete support. Our method avoids a sampling problem in mean-field variational inference (MFVI) caused by the so-called ’soap-bubble’ pathology of multivariate Gaussians. We show that, unlike MFVI, Radial BNNs are robust to hyperparameters and can be efficiently applied to challenging real-world tasks without needing ad-hoc tweaks and intensive tuning: on a real-world medical imaging task Radial BNNs outperform MC dropout and deep ensembles.

Cite this Paper

BibTeX
@InProceedings{pmlr-v108-farquhar20a, title = {Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning}, author = {Farquhar, Sebastian and Osborne, Michael A. and Gal, Yarin}, booktitle = {Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics}, pages = {1352--1362}, year = {2020}, editor = {Chiappa, Silvia and Calandra, Roberto}, volume = {108}, series = {Proceedings of Machine Learning Research}, month = {26--28 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v108/farquhar20a/farquhar20a.pdf}, url = {https://proceedings.mlr.press/v108/farquhar20a.html}, abstract = {We propose Radial Bayesian Neural Networks (BNNs): a variational approximate posterior for BNNs which scales well to large models. Unlike scalable Bayesian deep learning methods like deep ensembles that have discrete support (assign exactly zero probability almost everywhere in weight-space) Radial BNNs maintain full support; letting them act as a prior for continual learning and avoiding the a priori implausibility of discrete support. Our method avoids a sampling problem in mean-field variational inference (MFVI) caused by the so-called ’soap-bubble’ pathology of multivariate Gaussians. We show that, unlike MFVI, Radial BNNs are robust to hyperparameters and can be efficiently applied to challenging real-world tasks without needing ad-hoc tweaks and intensive tuning: on a real-world medical imaging task Radial BNNs outperform MC dropout and deep ensembles.} }
Endnote
%0 Conference Paper %T Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning %A Sebastian Farquhar %A Michael A. Osborne %A Yarin Gal %B Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2020 %E Silvia Chiappa %E Roberto Calandra %F pmlr-v108-farquhar20a %I PMLR %P 1352--1362 %U https://proceedings.mlr.press/v108/farquhar20a.html %V 108 %X We propose Radial Bayesian Neural Networks (BNNs): a variational approximate posterior for BNNs which scales well to large models. Unlike scalable Bayesian deep learning methods like deep ensembles that have discrete support (assign exactly zero probability almost everywhere in weight-space) Radial BNNs maintain full support; letting them act as a prior for continual learning and avoiding the a priori implausibility of discrete support. Our method avoids a sampling problem in mean-field variational inference (MFVI) caused by the so-called ’soap-bubble’ pathology of multivariate Gaussians. We show that, unlike MFVI, Radial BNNs are robust to hyperparameters and can be efficiently applied to challenging real-world tasks without needing ad-hoc tweaks and intensive tuning: on a real-world medical imaging task Radial BNNs outperform MC dropout and deep ensembles.
APA
Farquhar, S., Osborne, M.A. & Gal, Y.. (2020). Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 108:1352-1362 Available from https://proceedings.mlr.press/v108/farquhar20a.html.

Related Material