Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks

Jose Miguel Hernandez-Lobato, Ryan Adams
Proceedings of the 32nd International Conference on Machine Learning, PMLR 37:1861-1869, 2015.

Abstract

Large multilayer neural networks trained with backpropagation have recently achieved state-of-the-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic predictions, and a tendency to overfit the training data. In principle, the Bayesian approach to learning neural networks does not have these problems. However, existing Bayesian techniques lack scalability to large dataset and network sizes. In this work we present a novel scalable method for learning Bayesian neural networks, called probabilistic backpropagation (PBP). Similar to classical backpropagation, PBP works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients. A series of experiments on ten real-world datasets show that PBP is significantly faster than other techniques, while offering competitive predictive abilities. Our experiments also show that PBP provides accurate estimates of the posterior variance on the network weights.

Cite this Paper

BibTeX
@InProceedings{pmlr-v37-hernandez-lobatoc15, title = {Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks}, author = {Hernandez-Lobato, Jose Miguel and Adams, Ryan}, booktitle = {Proceedings of the 32nd International Conference on Machine Learning}, pages = {1861--1869}, year = {2015}, editor = {Bach, Francis and Blei, David}, volume = {37}, series = {Proceedings of Machine Learning Research}, address = {Lille, France}, month = {07--09 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v37/hernandez-lobatoc15.pdf}, url = {https://proceedings.mlr.press/v37/hernandez-lobatoc15.html}, abstract = {Large multilayer neural networks trained with backpropagation have recently achieved state-of-the-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic predictions, and a tendency to overfit the training data. In principle, the Bayesian approach to learning neural networks does not have these problems. However, existing Bayesian techniques lack scalability to large dataset and network sizes. In this work we present a novel scalable method for learning Bayesian neural networks, called probabilistic backpropagation (PBP). Similar to classical backpropagation, PBP works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients. A series of experiments on ten real-world datasets show that PBP is significantly faster than other techniques, while offering competitive predictive abilities. Our experiments also show that PBP provides accurate estimates of the posterior variance on the network weights.} }
Endnote
%0 Conference Paper %T Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks %A Jose Miguel Hernandez-Lobato %A Ryan Adams %B Proceedings of the 32nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2015 %E Francis Bach %E David Blei %F pmlr-v37-hernandez-lobatoc15 %I PMLR %P 1861--1869 %U https://proceedings.mlr.press/v37/hernandez-lobatoc15.html %V 37 %X Large multilayer neural networks trained with backpropagation have recently achieved state-of-the-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic predictions, and a tendency to overfit the training data. In principle, the Bayesian approach to learning neural networks does not have these problems. However, existing Bayesian techniques lack scalability to large dataset and network sizes. In this work we present a novel scalable method for learning Bayesian neural networks, called probabilistic backpropagation (PBP). Similar to classical backpropagation, PBP works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients. A series of experiments on ten real-world datasets show that PBP is significantly faster than other techniques, while offering competitive predictive abilities. Our experiments also show that PBP provides accurate estimates of the posterior variance on the network weights.
RIS
TY - CPAPER TI - Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks AU - Jose Miguel Hernandez-Lobato AU - Ryan Adams BT - Proceedings of the 32nd International Conference on Machine Learning DA - 2015/06/01 ED - Francis Bach ED - David Blei ID - pmlr-v37-hernandez-lobatoc15 PB - PMLR DP - Proceedings of Machine Learning Research VL - 37 SP - 1861 EP - 1869 L1 - http://proceedings.mlr.press/v37/hernandez-lobatoc15.pdf UR - https://proceedings.mlr.press/v37/hernandez-lobatoc15.html AB - Large multilayer neural networks trained with backpropagation have recently achieved state-of-the-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic predictions, and a tendency to overfit the training data. In principle, the Bayesian approach to learning neural networks does not have these problems. However, existing Bayesian techniques lack scalability to large dataset and network sizes. In this work we present a novel scalable method for learning Bayesian neural networks, called probabilistic backpropagation (PBP). Similar to classical backpropagation, PBP works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients. A series of experiments on ten real-world datasets show that PBP is significantly faster than other techniques, while offering competitive predictive abilities. Our experiments also show that PBP provides accurate estimates of the posterior variance on the network weights. ER -
APA
Hernandez-Lobato, J.M. & Adams, R.. (2015). Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks. Proceedings of the 32nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 37:1861-1869 Available from https://proceedings.mlr.press/v37/hernandez-lobatoc15.html.

Related Material