Deep Gaussian Processes for Regression using Approximate Expectation Propagation

Thang Bui, Daniel Hernandez-Lobato, Jose Hernandez-Lobato, Yingzhen Li, Richard Turner
Proceedings of The 33rd International Conference on Machine Learning, PMLR 48:1472-1481, 2016.

Abstract

Deep Gaussian processes (DGPs) are multi-layer hierarchical generalisations of Gaussian processes (GPs) and are formally equivalent to neural networks with multiple, infinitely wide hidden layers. DGPs are nonparametric probabilistic models and as such are arguably more flexible, have a greater capacity to generalise, and provide better calibrated uncertainty estimates than alternative deep models. This paper develops a new approximate Bayesian learning scheme that enables DGPs to be applied to a range of medium to large scale regression problems for the first time. The new method uses an approximate Expectation Propagation procedure and a novel and efficient extension of the probabilistic backpropagation algorithm for learning. We evaluate the new method for non-linear regression on eleven real-world datasets, showing that it always outperforms GP regression and is almost always better than state-of-the-art deterministic and sampling-based approximate inference methods for Bayesian neural networks. As a by-product, this work provides a comprehensive analysis of six approximate Bayesian methods for training neural networks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v48-bui16, title = {Deep Gaussian Processes for Regression using Approximate Expectation Propagation}, author = {Bui, Thang and Hernandez-Lobato, Daniel and Hernandez-Lobato, Jose and Li, Yingzhen and Turner, Richard}, booktitle = {Proceedings of The 33rd International Conference on Machine Learning}, pages = {1472--1481}, year = {2016}, editor = {Balcan, Maria Florina and Weinberger, Kilian Q.}, volume = {48}, series = {Proceedings of Machine Learning Research}, address = {New York, New York, USA}, month = {20--22 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v48/bui16.pdf}, url = {https://proceedings.mlr.press/v48/bui16.html}, abstract = {Deep Gaussian processes (DGPs) are multi-layer hierarchical generalisations of Gaussian processes (GPs) and are formally equivalent to neural networks with multiple, infinitely wide hidden layers. DGPs are nonparametric probabilistic models and as such are arguably more flexible, have a greater capacity to generalise, and provide better calibrated uncertainty estimates than alternative deep models. This paper develops a new approximate Bayesian learning scheme that enables DGPs to be applied to a range of medium to large scale regression problems for the first time. The new method uses an approximate Expectation Propagation procedure and a novel and efficient extension of the probabilistic backpropagation algorithm for learning. We evaluate the new method for non-linear regression on eleven real-world datasets, showing that it always outperforms GP regression and is almost always better than state-of-the-art deterministic and sampling-based approximate inference methods for Bayesian neural networks. As a by-product, this work provides a comprehensive analysis of six approximate Bayesian methods for training neural networks.} }
Endnote
%0 Conference Paper %T Deep Gaussian Processes for Regression using Approximate Expectation Propagation %A Thang Bui %A Daniel Hernandez-Lobato %A Jose Hernandez-Lobato %A Yingzhen Li %A Richard Turner %B Proceedings of The 33rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2016 %E Maria Florina Balcan %E Kilian Q. Weinberger %F pmlr-v48-bui16 %I PMLR %P 1472--1481 %U https://proceedings.mlr.press/v48/bui16.html %V 48 %X Deep Gaussian processes (DGPs) are multi-layer hierarchical generalisations of Gaussian processes (GPs) and are formally equivalent to neural networks with multiple, infinitely wide hidden layers. DGPs are nonparametric probabilistic models and as such are arguably more flexible, have a greater capacity to generalise, and provide better calibrated uncertainty estimates than alternative deep models. This paper develops a new approximate Bayesian learning scheme that enables DGPs to be applied to a range of medium to large scale regression problems for the first time. The new method uses an approximate Expectation Propagation procedure and a novel and efficient extension of the probabilistic backpropagation algorithm for learning. We evaluate the new method for non-linear regression on eleven real-world datasets, showing that it always outperforms GP regression and is almost always better than state-of-the-art deterministic and sampling-based approximate inference methods for Bayesian neural networks. As a by-product, this work provides a comprehensive analysis of six approximate Bayesian methods for training neural networks.
RIS
TY - CPAPER TI - Deep Gaussian Processes for Regression using Approximate Expectation Propagation AU - Thang Bui AU - Daniel Hernandez-Lobato AU - Jose Hernandez-Lobato AU - Yingzhen Li AU - Richard Turner BT - Proceedings of The 33rd International Conference on Machine Learning DA - 2016/06/11 ED - Maria Florina Balcan ED - Kilian Q. Weinberger ID - pmlr-v48-bui16 PB - PMLR DP - Proceedings of Machine Learning Research VL - 48 SP - 1472 EP - 1481 L1 - http://proceedings.mlr.press/v48/bui16.pdf UR - https://proceedings.mlr.press/v48/bui16.html AB - Deep Gaussian processes (DGPs) are multi-layer hierarchical generalisations of Gaussian processes (GPs) and are formally equivalent to neural networks with multiple, infinitely wide hidden layers. DGPs are nonparametric probabilistic models and as such are arguably more flexible, have a greater capacity to generalise, and provide better calibrated uncertainty estimates than alternative deep models. This paper develops a new approximate Bayesian learning scheme that enables DGPs to be applied to a range of medium to large scale regression problems for the first time. The new method uses an approximate Expectation Propagation procedure and a novel and efficient extension of the probabilistic backpropagation algorithm for learning. We evaluate the new method for non-linear regression on eleven real-world datasets, showing that it always outperforms GP regression and is almost always better than state-of-the-art deterministic and sampling-based approximate inference methods for Bayesian neural networks. As a by-product, this work provides a comprehensive analysis of six approximate Bayesian methods for training neural networks. ER -
APA
Bui, T., Hernandez-Lobato, D., Hernandez-Lobato, J., Li, Y. & Turner, R.. (2016). Deep Gaussian Processes for Regression using Approximate Expectation Propagation. Proceedings of The 33rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 48:1472-1481 Available from https://proceedings.mlr.press/v48/bui16.html.

Related Material