Adaptive Cholesky Gaussian Processes

Simon Bartels, Kristoffer Stensbo-Smidt, Pablo Moreno-Munoz, Wouter Boomsma, Jes Frellsen, Soren Hauberg
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:408-452, 2023.

Abstract

We present a method to approximate Gaussian process regression models to large datasets by considering only a subset of the data. Our approach is novel in that the size of the subset is selected on the fly during exact inference with little computational overhead. From an empirical observation that the log-marginal likelihood often exhibits a linear trend once a sufficient subset of a dataset has been observed, we conclude that many large datasets contain redundant information that only slightly affects the posterior. Based on this, we provide probabilistic bounds on the full model evidence that can identify such subsets. Remarkably, these bounds are largely composed of terms that appear in intermediate steps of the standard Cholesky decomposition, allowing us to modify the algorithm to adaptively stop the decomposition once enough data have been observed.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-bartels23a, title = {Adaptive Cholesky Gaussian Processes}, author = {Bartels, Simon and Stensbo-Smidt, Kristoffer and Moreno-Munoz, Pablo and Boomsma, Wouter and Frellsen, Jes and Hauberg, Soren}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {408--452}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/bartels23a/bartels23a.pdf}, url = {https://proceedings.mlr.press/v206/bartels23a.html}, abstract = {We present a method to approximate Gaussian process regression models to large datasets by considering only a subset of the data. Our approach is novel in that the size of the subset is selected on the fly during exact inference with little computational overhead. From an empirical observation that the log-marginal likelihood often exhibits a linear trend once a sufficient subset of a dataset has been observed, we conclude that many large datasets contain redundant information that only slightly affects the posterior. Based on this, we provide probabilistic bounds on the full model evidence that can identify such subsets. Remarkably, these bounds are largely composed of terms that appear in intermediate steps of the standard Cholesky decomposition, allowing us to modify the algorithm to adaptively stop the decomposition once enough data have been observed.} }
Endnote
%0 Conference Paper %T Adaptive Cholesky Gaussian Processes %A Simon Bartels %A Kristoffer Stensbo-Smidt %A Pablo Moreno-Munoz %A Wouter Boomsma %A Jes Frellsen %A Soren Hauberg %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-bartels23a %I PMLR %P 408--452 %U https://proceedings.mlr.press/v206/bartels23a.html %V 206 %X We present a method to approximate Gaussian process regression models to large datasets by considering only a subset of the data. Our approach is novel in that the size of the subset is selected on the fly during exact inference with little computational overhead. From an empirical observation that the log-marginal likelihood often exhibits a linear trend once a sufficient subset of a dataset has been observed, we conclude that many large datasets contain redundant information that only slightly affects the posterior. Based on this, we provide probabilistic bounds on the full model evidence that can identify such subsets. Remarkably, these bounds are largely composed of terms that appear in intermediate steps of the standard Cholesky decomposition, allowing us to modify the algorithm to adaptively stop the decomposition once enough data have been observed.
APA
Bartels, S., Stensbo-Smidt, K., Moreno-Munoz, P., Boomsma, W., Frellsen, J. & Hauberg, S.. (2023). Adaptive Cholesky Gaussian Processes. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:408-452 Available from https://proceedings.mlr.press/v206/bartels23a.html.

Related Material