Actually Sparse Variational Gaussian Processes

Harry Jake Cunningham, Daniel Augusto de Souza, So Takao, Mark van der Wilk, Marc Peter Deisenroth
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:10395-10408, 2023.

Abstract

Gaussian processes (GPs) are typically criticised for their unfavourable scaling in both computational and memory requirements. For large datasets, sparse GPs reduce these demands by conditioning on a small set of inducing variables designed to summarise the data. In practice however, for large datasets requiring many inducing variables, such as low-lengthscale spatial data, even sparse GPs can become computationally expensive, limited by the number of inducing variables one can use. In this work, we propose a new class of inter-domain variational GP, constructed by projecting a GP onto a set of compactly supported B-spline basis functions. The key benefit of our approach is that the compact support of the B-spline basis functions admits the use of sparse linear algebra to significantly speed up matrix operations and drastically reduce the memory footprint. This allows us to very efficiently model fast-varying spatial phenomena with tens of thousands of inducing variables, where previous approaches failed.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-cunningham23a, title = {Actually Sparse Variational Gaussian Processes}, author = {Cunningham, Harry Jake and de Souza, Daniel Augusto and Takao, So and van der Wilk, Mark and Deisenroth, Marc Peter}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {10395--10408}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/cunningham23a/cunningham23a.pdf}, url = {https://proceedings.mlr.press/v206/cunningham23a.html}, abstract = {Gaussian processes (GPs) are typically criticised for their unfavourable scaling in both computational and memory requirements. For large datasets, sparse GPs reduce these demands by conditioning on a small set of inducing variables designed to summarise the data. In practice however, for large datasets requiring many inducing variables, such as low-lengthscale spatial data, even sparse GPs can become computationally expensive, limited by the number of inducing variables one can use. In this work, we propose a new class of inter-domain variational GP, constructed by projecting a GP onto a set of compactly supported B-spline basis functions. The key benefit of our approach is that the compact support of the B-spline basis functions admits the use of sparse linear algebra to significantly speed up matrix operations and drastically reduce the memory footprint. This allows us to very efficiently model fast-varying spatial phenomena with tens of thousands of inducing variables, where previous approaches failed.} }
Endnote
%0 Conference Paper %T Actually Sparse Variational Gaussian Processes %A Harry Jake Cunningham %A Daniel Augusto de Souza %A So Takao %A Mark van der Wilk %A Marc Peter Deisenroth %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-cunningham23a %I PMLR %P 10395--10408 %U https://proceedings.mlr.press/v206/cunningham23a.html %V 206 %X Gaussian processes (GPs) are typically criticised for their unfavourable scaling in both computational and memory requirements. For large datasets, sparse GPs reduce these demands by conditioning on a small set of inducing variables designed to summarise the data. In practice however, for large datasets requiring many inducing variables, such as low-lengthscale spatial data, even sparse GPs can become computationally expensive, limited by the number of inducing variables one can use. In this work, we propose a new class of inter-domain variational GP, constructed by projecting a GP onto a set of compactly supported B-spline basis functions. The key benefit of our approach is that the compact support of the B-spline basis functions admits the use of sparse linear algebra to significantly speed up matrix operations and drastically reduce the memory footprint. This allows us to very efficiently model fast-varying spatial phenomena with tens of thousands of inducing variables, where previous approaches failed.
APA
Cunningham, H.J., de Souza, D.A., Takao, S., van der Wilk, M. & Deisenroth, M.P.. (2023). Actually Sparse Variational Gaussian Processes. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:10395-10408 Available from https://proceedings.mlr.press/v206/cunningham23a.html.

Related Material