Fast Gaussian process regression for high dimensional functions with derivative information

Aleksei Sorokin, Pieterjan Robbe, Fred J Hickernell
Proceedings of the First International Conference on Probabilistic Numerics, PMLR 271:35-49, 2025.

Abstract

Gaussian process regression (GPR) is the backbone of many methods in probabilistic numerics. Exact GPR on $n$ sampling locations in $d$ dimensions generally requires $\mathcal{O}(n^2)$ storage and costs $\mathcal{O}(n^3+dn^2)$ to fit assuming kernel evaluation costs $\mathcal{O}(d)$. Using certain pairings of sampling locations and kernels induces nice structure into the Gram matrix and enables accelerated exact GPR. One popular pairing uses Cartesian grids with product kernels to induce Kronecker structure in the Gram matrix. This reduces exact GP fitting costs to $\mathcal{O}(d n^{3/d})$ and storage requirements to $\mathcal{O}(d n^{2/d})$, but quickly becomes intractable when the dimension exceeds a few dozen. Recent work has shown that pairings certain low discrepancy sequences with special kernels enables GPR to scale like $\mathcal{O}(n \log n + nd)$ for fitting costs and $\mathcal{O}(n)$ for storage requirements. We describe an extension of these methods to problems which observe $m$ derivative multi-indices at each of the $n$ low discrepancy sampling locations. By exploiting similar structure across blocks of the Gram matrix, we are able to reduce the GP fitting cost from $\mathcal{O}(n^3 m^3 + n^2m^2d)$ to $\mathcal{O}(m^2 n \log n+m^3n + n m^2 d)$ and reduce the storage requirements from $\mathcal{O}(n^2 m^2)$ to $\mathcal{O}(nm^2)$. We explore a number of synthetic benchmarks to illustrate the potential of the proposed approach.

Cite this Paper

BibTeX
@InProceedings{pmlr-v271-sorokin25a, title = {Fast {G}aussian process regression for high dimensional functions with derivative information}, author = {Sorokin, Aleksei and Robbe, Pieterjan and Hickernell, Fred J}, booktitle = {Proceedings of the First International Conference on Probabilistic Numerics}, pages = {35--49}, year = {2025}, editor = {Kanagawa, Motonobu and Cockayne, Jon and Gessner, Alexandra and Hennig, Philipp}, volume = {271}, series = {Proceedings of Machine Learning Research}, month = {01--03 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v271/main/assets/sorokin25a/sorokin25a.pdf}, url = {https://proceedings.mlr.press/v271/sorokin25a.html}, abstract = {Gaussian process regression (GPR) is the backbone of many methods in probabilistic numerics. Exact GPR on $n$ sampling locations in $d$ dimensions generally requires $\mathcal{O}(n^2)$ storage and costs $\mathcal{O}(n^3+dn^2)$ to fit assuming kernel evaluation costs $\mathcal{O}(d)$. Using certain pairings of sampling locations and kernels induces nice structure into the Gram matrix and enables accelerated exact GPR. One popular pairing uses Cartesian grids with product kernels to induce Kronecker structure in the Gram matrix. This reduces exact GP fitting costs to $\mathcal{O}(d n^{3/d})$ and storage requirements to $\mathcal{O}(d n^{2/d})$, but quickly becomes intractable when the dimension exceeds a few dozen. Recent work has shown that pairings certain low discrepancy sequences with special kernels enables GPR to scale like $\mathcal{O}(n \log n + nd)$ for fitting costs and $\mathcal{O}(n)$ for storage requirements. We describe an extension of these methods to problems which observe $m$ derivative multi-indices at each of the $n$ low discrepancy sampling locations. By exploiting similar structure across blocks of the Gram matrix, we are able to reduce the GP fitting cost from $\mathcal{O}(n^3 m^3 + n^2m^2d)$ to $\mathcal{O}(m^2 n \log n+m^3n + n m^2 d)$ and reduce the storage requirements from $\mathcal{O}(n^2 m^2)$ to $\mathcal{O}(nm^2)$. We explore a number of synthetic benchmarks to illustrate the potential of the proposed approach.} }
Endnote
%0 Conference Paper %T Fast Gaussian process regression for high dimensional functions with derivative information %A Aleksei Sorokin %A Pieterjan Robbe %A Fred J Hickernell %B Proceedings of the First International Conference on Probabilistic Numerics %C Proceedings of Machine Learning Research %D 2025 %E Motonobu Kanagawa %E Jon Cockayne %E Alexandra Gessner %E Philipp Hennig %F pmlr-v271-sorokin25a %I PMLR %P 35--49 %U https://proceedings.mlr.press/v271/sorokin25a.html %V 271 %X Gaussian process regression (GPR) is the backbone of many methods in probabilistic numerics. Exact GPR on $n$ sampling locations in $d$ dimensions generally requires $\mathcal{O}(n^2)$ storage and costs $\mathcal{O}(n^3+dn^2)$ to fit assuming kernel evaluation costs $\mathcal{O}(d)$. Using certain pairings of sampling locations and kernels induces nice structure into the Gram matrix and enables accelerated exact GPR. One popular pairing uses Cartesian grids with product kernels to induce Kronecker structure in the Gram matrix. This reduces exact GP fitting costs to $\mathcal{O}(d n^{3/d})$ and storage requirements to $\mathcal{O}(d n^{2/d})$, but quickly becomes intractable when the dimension exceeds a few dozen. Recent work has shown that pairings certain low discrepancy sequences with special kernels enables GPR to scale like $\mathcal{O}(n \log n + nd)$ for fitting costs and $\mathcal{O}(n)$ for storage requirements. We describe an extension of these methods to problems which observe $m$ derivative multi-indices at each of the $n$ low discrepancy sampling locations. By exploiting similar structure across blocks of the Gram matrix, we are able to reduce the GP fitting cost from $\mathcal{O}(n^3 m^3 + n^2m^2d)$ to $\mathcal{O}(m^2 n \log n+m^3n + n m^2 d)$ and reduce the storage requirements from $\mathcal{O}(n^2 m^2)$ to $\mathcal{O}(nm^2)$. We explore a number of synthetic benchmarks to illustrate the potential of the proposed approach.
APA
Sorokin, A., Robbe, P. & Hickernell, F.J.. (2025). Fast Gaussian process regression for high dimensional functions with derivative information. Proceedings of the First International Conference on Probabilistic Numerics, in Proceedings of Machine Learning Research 271:35-49 Available from https://proceedings.mlr.press/v271/sorokin25a.html.

Related Material