Bayesian Uncertainty Estimation for Batch Normalized Deep Networks

Mattias Teye, Hossein Azizpour, Kevin Smith
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:4907-4916, 2018.

Abstract

We show that training a deep network using batch normalization is equivalent to approximate inference in Bayesian models. We further demonstrate that this finding allows us to make meaningful estimates of the model uncertainty using conventional architectures, without modifications to the network or the training procedure. Our approach is thoroughly validated by measuring the quality of uncertainty in a series of empirical experiments on different tasks. It outperforms baselines with strong statistical significance, and displays competitive performance with recent Bayesian approaches.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-teye18a, title = {{B}ayesian Uncertainty Estimation for Batch Normalized Deep Networks}, author = {Teye, Mattias and Azizpour, Hossein and Smith, Kevin}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {4907--4916}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/teye18a/teye18a.pdf}, url = {https://proceedings.mlr.press/v80/teye18a.html}, abstract = {We show that training a deep network using batch normalization is equivalent to approximate inference in Bayesian models. We further demonstrate that this finding allows us to make meaningful estimates of the model uncertainty using conventional architectures, without modifications to the network or the training procedure. Our approach is thoroughly validated by measuring the quality of uncertainty in a series of empirical experiments on different tasks. It outperforms baselines with strong statistical significance, and displays competitive performance with recent Bayesian approaches.} }
Endnote
%0 Conference Paper %T Bayesian Uncertainty Estimation for Batch Normalized Deep Networks %A Mattias Teye %A Hossein Azizpour %A Kevin Smith %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-teye18a %I PMLR %P 4907--4916 %U https://proceedings.mlr.press/v80/teye18a.html %V 80 %X We show that training a deep network using batch normalization is equivalent to approximate inference in Bayesian models. We further demonstrate that this finding allows us to make meaningful estimates of the model uncertainty using conventional architectures, without modifications to the network or the training procedure. Our approach is thoroughly validated by measuring the quality of uncertainty in a series of empirical experiments on different tasks. It outperforms baselines with strong statistical significance, and displays competitive performance with recent Bayesian approaches.
APA
Teye, M., Azizpour, H. & Smith, K.. (2018). Bayesian Uncertainty Estimation for Batch Normalized Deep Networks. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:4907-4916 Available from https://proceedings.mlr.press/v80/teye18a.html.

Related Material