BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuning

Zhijie Deng, Jun Zhu
Proceedings of The 14th Asian Conference on Machine Learning, PMLR 189:280-295, 2023.

Abstract

Despite their theoretical appealingness, Bayesian neural networks (BNNs) are left behind in real-world adoption, mainly due to persistent concerns on their scalability, accessibility, and reliability. In this work, we develop the BayesAdapter framework to relieve these concerns. In particular, we propose to adapt pre-trained deterministic NNs to be variational BNNs via cost-effective Bayesian fine-tuning. Technically, we develop a modularized implementation for the learning of variational BNNs, and refurbish the generally applicable exemplar reparameterization trick through exemplar parallelization to efficiently reduce the gradient variance in stochastic variational inference. Based on the the lightweight Bayesian learning paradigm, we conduct extensive experiments on a variety of benchmarks, and show that our method can consistently induce posteriors with higher quality than competitive baselines, yet significantly reducing training overheads.

Cite this Paper

BibTeX
@InProceedings{pmlr-v189-deng23b, title = {BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuning}, author = {Deng, Zhijie and Zhu, Jun}, booktitle = {Proceedings of The 14th Asian Conference on Machine Learning}, pages = {280--295}, year = {2023}, editor = {Khan, Emtiyaz and Gonen, Mehmet}, volume = {189}, series = {Proceedings of Machine Learning Research}, month = {12--14 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v189/deng23b/deng23b.pdf}, url = {https://proceedings.mlr.press/v189/deng23b.html}, abstract = {Despite their theoretical appealingness, Bayesian neural networks (BNNs) are left behind in real-world adoption, mainly due to persistent concerns on their scalability, accessibility, and reliability. In this work, we develop the BayesAdapter framework to relieve these concerns. In particular, we propose to adapt pre-trained deterministic NNs to be variational BNNs via cost-effective Bayesian fine-tuning. Technically, we develop a modularized implementation for the learning of variational BNNs, and refurbish the generally applicable exemplar reparameterization trick through exemplar parallelization to efficiently reduce the gradient variance in stochastic variational inference. Based on the the lightweight Bayesian learning paradigm, we conduct extensive experiments on a variety of benchmarks, and show that our method can consistently induce posteriors with higher quality than competitive baselines, yet significantly reducing training overheads.} }
Endnote
%0 Conference Paper %T BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuning %A Zhijie Deng %A Jun Zhu %B Proceedings of The 14th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Emtiyaz Khan %E Mehmet Gonen %F pmlr-v189-deng23b %I PMLR %P 280--295 %U https://proceedings.mlr.press/v189/deng23b.html %V 189 %X Despite their theoretical appealingness, Bayesian neural networks (BNNs) are left behind in real-world adoption, mainly due to persistent concerns on their scalability, accessibility, and reliability. In this work, we develop the BayesAdapter framework to relieve these concerns. In particular, we propose to adapt pre-trained deterministic NNs to be variational BNNs via cost-effective Bayesian fine-tuning. Technically, we develop a modularized implementation for the learning of variational BNNs, and refurbish the generally applicable exemplar reparameterization trick through exemplar parallelization to efficiently reduce the gradient variance in stochastic variational inference. Based on the the lightweight Bayesian learning paradigm, we conduct extensive experiments on a variety of benchmarks, and show that our method can consistently induce posteriors with higher quality than competitive baselines, yet significantly reducing training overheads.
APA
Deng, Z. & Zhu, J.. (2023). BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuning. Proceedings of The 14th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 189:280-295 Available from https://proceedings.mlr.press/v189/deng23b.html.

Related Material