Healing Products of Gaussian Process Experts

Samuel Cohen, Rendani Mbuvha, Tshilidzi Marwala, Marc Deisenroth
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:2068-2077, 2020.

Abstract

Gaussian processes (GPs) are nonparametric Bayesian models that have been applied to regression and classification problems. One of the approaches to alleviate their cubic training cost is the use of local GP experts trained on subsets of the data. In particular, product-of-expert models combine the predictive distributions of local experts through a tractable product operation. While these expert models allow for massively distributed computation, their predictions typically suffer from erratic behaviour of the mean or uncalibrated uncertainty quantification. By calibrating predictions via a tempered softmax weighting, we provide a solution to these problems for multiple product-of-expert models, including the generalised product of experts and the robust Bayesian committee machine. Furthermore, we leverage the optimal transport literature and propose a new product-of-expert model that combines predictions of local experts by computing their Wasserstein barycenter, which can be applied to both regression and classification.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-cohen20b, title = {Healing Products of {G}aussian Process Experts}, author = {Cohen, Samuel and Mbuvha, Rendani and Marwala, Tshilidzi and Deisenroth, Marc}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {2068--2077}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/cohen20b/cohen20b.pdf}, url = {https://proceedings.mlr.press/v119/cohen20b.html}, abstract = {Gaussian processes (GPs) are nonparametric Bayesian models that have been applied to regression and classification problems. One of the approaches to alleviate their cubic training cost is the use of local GP experts trained on subsets of the data. In particular, product-of-expert models combine the predictive distributions of local experts through a tractable product operation. While these expert models allow for massively distributed computation, their predictions typically suffer from erratic behaviour of the mean or uncalibrated uncertainty quantification. By calibrating predictions via a tempered softmax weighting, we provide a solution to these problems for multiple product-of-expert models, including the generalised product of experts and the robust Bayesian committee machine. Furthermore, we leverage the optimal transport literature and propose a new product-of-expert model that combines predictions of local experts by computing their Wasserstein barycenter, which can be applied to both regression and classification.} }
Endnote
%0 Conference Paper %T Healing Products of Gaussian Process Experts %A Samuel Cohen %A Rendani Mbuvha %A Tshilidzi Marwala %A Marc Deisenroth %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-cohen20b %I PMLR %P 2068--2077 %U https://proceedings.mlr.press/v119/cohen20b.html %V 119 %X Gaussian processes (GPs) are nonparametric Bayesian models that have been applied to regression and classification problems. One of the approaches to alleviate their cubic training cost is the use of local GP experts trained on subsets of the data. In particular, product-of-expert models combine the predictive distributions of local experts through a tractable product operation. While these expert models allow for massively distributed computation, their predictions typically suffer from erratic behaviour of the mean or uncalibrated uncertainty quantification. By calibrating predictions via a tempered softmax weighting, we provide a solution to these problems for multiple product-of-expert models, including the generalised product of experts and the robust Bayesian committee machine. Furthermore, we leverage the optimal transport literature and propose a new product-of-expert model that combines predictions of local experts by computing their Wasserstein barycenter, which can be applied to both regression and classification.
APA
Cohen, S., Mbuvha, R., Marwala, T. & Deisenroth, M.. (2020). Healing Products of Gaussian Process Experts. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:2068-2077 Available from https://proceedings.mlr.press/v119/cohen20b.html.

Related Material