Scalable Multi-Class Gaussian Process Classification using Expectation Propagation

Carlos Villacampa-Calvo, Daniel Hernández-Lobato
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:3550-3559, 2017.

Abstract

This paper describes an expectation propagation (EP) method for multi-class classification with Gaussian processes that scales well to very large datasets. In such a method the estimate of the log-marginal-likelihood involves a sum across the data instances. This enables efficient training using stochastic gradients and mini-batches. When this type of training is used, the computational cost does not depend on the number of data instances N. Furthermore, extra assumptions in the approximate inference process make the memory cost independent of N. The consequence is that the proposed EP method can be used on datasets with millions of instances. We compare empirically this method with alternative approaches that approximate the required computations using variational inference. The results show that it performs similar or even better than these techniques, which sometimes give significantly worse predictive distributions in terms of the test log-likelihood. Besides this, the training process of the proposed approach also seems to converge in a smaller number of iterations.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-villacampa-calvo17a, title = {Scalable Multi-Class {G}aussian Process Classification using Expectation Propagation}, author = {Carlos Villacampa-Calvo and Daniel Hern{\'a}ndez-Lobato}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {3550--3559}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/villacampa-calvo17a/villacampa-calvo17a.pdf}, url = {https://proceedings.mlr.press/v70/villacampa-calvo17a.html}, abstract = {This paper describes an expectation propagation (EP) method for multi-class classification with Gaussian processes that scales well to very large datasets. In such a method the estimate of the log-marginal-likelihood involves a sum across the data instances. This enables efficient training using stochastic gradients and mini-batches. When this type of training is used, the computational cost does not depend on the number of data instances N. Furthermore, extra assumptions in the approximate inference process make the memory cost independent of N. The consequence is that the proposed EP method can be used on datasets with millions of instances. We compare empirically this method with alternative approaches that approximate the required computations using variational inference. The results show that it performs similar or even better than these techniques, which sometimes give significantly worse predictive distributions in terms of the test log-likelihood. Besides this, the training process of the proposed approach also seems to converge in a smaller number of iterations.} }
Endnote
%0 Conference Paper %T Scalable Multi-Class Gaussian Process Classification using Expectation Propagation %A Carlos Villacampa-Calvo %A Daniel Hernández-Lobato %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-villacampa-calvo17a %I PMLR %P 3550--3559 %U https://proceedings.mlr.press/v70/villacampa-calvo17a.html %V 70 %X This paper describes an expectation propagation (EP) method for multi-class classification with Gaussian processes that scales well to very large datasets. In such a method the estimate of the log-marginal-likelihood involves a sum across the data instances. This enables efficient training using stochastic gradients and mini-batches. When this type of training is used, the computational cost does not depend on the number of data instances N. Furthermore, extra assumptions in the approximate inference process make the memory cost independent of N. The consequence is that the proposed EP method can be used on datasets with millions of instances. We compare empirically this method with alternative approaches that approximate the required computations using variational inference. The results show that it performs similar or even better than these techniques, which sometimes give significantly worse predictive distributions in terms of the test log-likelihood. Besides this, the training process of the proposed approach also seems to converge in a smaller number of iterations.
APA
Villacampa-Calvo, C. & Hernández-Lobato, D.. (2017). Scalable Multi-Class Gaussian Process Classification using Expectation Propagation. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:3550-3559 Available from https://proceedings.mlr.press/v70/villacampa-calvo17a.html.

Related Material