Sparse within Sparse Gaussian Processes using Neighbor Information

Gia-Lac Tran, Dimitrios Milios, Pietro Michiardi, Maurizio Filippone
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:10369-10378, 2021.

Abstract

Approximations to Gaussian processes (GPs) based on inducing variables, combined with variational inference techniques, enable state-of-the-art sparse approaches to infer GPs at scale through mini-batch based learning. In this work, we further push the limits of scalability of sparse GPs by allowing large number of inducing variables without imposing a special structure on the inducing inputs. In particular, we introduce a novel hierarchical prior, which imposes sparsity on the set of inducing variables. We treat our model variationally, and we experimentally show considerable computational gains compared to standard sparse GPs when sparsity on the inducing variables is realized considering the nearest inducing inputs of a random mini-batch of the data. We perform an extensive experimental validation that demonstrates the effectiveness of our approach compared to the state-of-the-art. Our approach enables the possibility to use sparse GPs using a large number of inducing points without incurring a prohibitive computational cost.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-tran21a, title = {Sparse within Sparse Gaussian Processes using Neighbor Information}, author = {Tran, Gia-Lac and Milios, Dimitrios and Michiardi, Pietro and Filippone, Maurizio}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {10369--10378}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/tran21a/tran21a.pdf}, url = {https://proceedings.mlr.press/v139/tran21a.html}, abstract = {Approximations to Gaussian processes (GPs) based on inducing variables, combined with variational inference techniques, enable state-of-the-art sparse approaches to infer GPs at scale through mini-batch based learning. In this work, we further push the limits of scalability of sparse GPs by allowing large number of inducing variables without imposing a special structure on the inducing inputs. In particular, we introduce a novel hierarchical prior, which imposes sparsity on the set of inducing variables. We treat our model variationally, and we experimentally show considerable computational gains compared to standard sparse GPs when sparsity on the inducing variables is realized considering the nearest inducing inputs of a random mini-batch of the data. We perform an extensive experimental validation that demonstrates the effectiveness of our approach compared to the state-of-the-art. Our approach enables the possibility to use sparse GPs using a large number of inducing points without incurring a prohibitive computational cost.} }
Endnote
%0 Conference Paper %T Sparse within Sparse Gaussian Processes using Neighbor Information %A Gia-Lac Tran %A Dimitrios Milios %A Pietro Michiardi %A Maurizio Filippone %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-tran21a %I PMLR %P 10369--10378 %U https://proceedings.mlr.press/v139/tran21a.html %V 139 %X Approximations to Gaussian processes (GPs) based on inducing variables, combined with variational inference techniques, enable state-of-the-art sparse approaches to infer GPs at scale through mini-batch based learning. In this work, we further push the limits of scalability of sparse GPs by allowing large number of inducing variables without imposing a special structure on the inducing inputs. In particular, we introduce a novel hierarchical prior, which imposes sparsity on the set of inducing variables. We treat our model variationally, and we experimentally show considerable computational gains compared to standard sparse GPs when sparsity on the inducing variables is realized considering the nearest inducing inputs of a random mini-batch of the data. We perform an extensive experimental validation that demonstrates the effectiveness of our approach compared to the state-of-the-art. Our approach enables the possibility to use sparse GPs using a large number of inducing points without incurring a prohibitive computational cost.
APA
Tran, G., Milios, D., Michiardi, P. & Filippone, M.. (2021). Sparse within Sparse Gaussian Processes using Neighbor Information. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:10369-10378 Available from https://proceedings.mlr.press/v139/tran21a.html.

Related Material