Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference

Insung Kong, Dongyoon Yang, Jongjin Lee, Ilsang Ohn, Gyuseung Baek, Yongdai Kim
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:17462-17491, 2023.

Abstract

Bayesian approaches for learning deep neural networks (BNN) have been received much attention and successfully applied to various applications. Particularly, BNNs have the merit of having better generalization ability as well as better uncertainty quantification. For the success of BNN, search an appropriate architecture of the neural networks is an important task, and various algorithms to find good sparse neural networks have been proposed. In this paper, we propose a new node-sparse BNN model which has good theoretical properties and is computationally feasible. We prove that the posterior concentration rate to the true model is near minimax optimal and adaptive to the smoothness of the true model. In particular the adaptiveness is the first of its kind for node-sparse BNNs. In addition, we develop a novel MCMC algorithm which makes the Bayesian inference of the node-sparse BNN model feasible in practice.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-kong23e, title = {Masked {B}ayesian Neural Networks : Theoretical Guarantee and its Posterior Inference}, author = {Kong, Insung and Yang, Dongyoon and Lee, Jongjin and Ohn, Ilsang and Baek, Gyuseung and Kim, Yongdai}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {17462--17491}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/kong23e/kong23e.pdf}, url = {https://proceedings.mlr.press/v202/kong23e.html}, abstract = {Bayesian approaches for learning deep neural networks (BNN) have been received much attention and successfully applied to various applications. Particularly, BNNs have the merit of having better generalization ability as well as better uncertainty quantification. For the success of BNN, search an appropriate architecture of the neural networks is an important task, and various algorithms to find good sparse neural networks have been proposed. In this paper, we propose a new node-sparse BNN model which has good theoretical properties and is computationally feasible. We prove that the posterior concentration rate to the true model is near minimax optimal and adaptive to the smoothness of the true model. In particular the adaptiveness is the first of its kind for node-sparse BNNs. In addition, we develop a novel MCMC algorithm which makes the Bayesian inference of the node-sparse BNN model feasible in practice.} }
Endnote
%0 Conference Paper %T Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference %A Insung Kong %A Dongyoon Yang %A Jongjin Lee %A Ilsang Ohn %A Gyuseung Baek %A Yongdai Kim %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-kong23e %I PMLR %P 17462--17491 %U https://proceedings.mlr.press/v202/kong23e.html %V 202 %X Bayesian approaches for learning deep neural networks (BNN) have been received much attention and successfully applied to various applications. Particularly, BNNs have the merit of having better generalization ability as well as better uncertainty quantification. For the success of BNN, search an appropriate architecture of the neural networks is an important task, and various algorithms to find good sparse neural networks have been proposed. In this paper, we propose a new node-sparse BNN model which has good theoretical properties and is computationally feasible. We prove that the posterior concentration rate to the true model is near minimax optimal and adaptive to the smoothness of the true model. In particular the adaptiveness is the first of its kind for node-sparse BNNs. In addition, we develop a novel MCMC algorithm which makes the Bayesian inference of the node-sparse BNN model feasible in practice.
APA
Kong, I., Yang, D., Lee, J., Ohn, I., Baek, G. & Kim, Y.. (2023). Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:17462-17491 Available from https://proceedings.mlr.press/v202/kong23e.html.

Related Material