Tensor-Train Kernel Learning for Gaussian Processes

Max Kirstein, David Sommer, Martin Eigel
Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 179:253-272, 2022.

Abstract

We propose a new kernel learning approach based on efficient low-rank tensor compression for Gaussian process (GP) regression. The central idea is to compose a low-rank function represented in a hierarchical tensor format with a GP covariance function. Compared to similar deep neural network architectures, this approach facilitates to learn significantly more expressive features at lower computational costs as illustrated in the examples. Additionally, over-fitting is avoided with this compositional model by taking advantage of its inherent regularisation properties. Estimates of the generalisation error are compared to five baseline models on three synthetic and six real-world data sets. The experimental results show that the incorporated tensor network enables a highly accurate GP regression with a comparatively low number of trainable parameters. The observed performance is clearly superior (usually by an order of magnitude in mean squared error) to all examined standard models, in particular to deep neural networks with more than 1000 times as many parameters.

Cite this Paper

BibTeX
@InProceedings{pmlr-v179-kirstein22a, title = {Tensor-Train Kernel Learning for Gaussian Processes}, author = {Kirstein, Max and Sommer, David and Eigel, Martin}, booktitle = {Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {253--272}, year = {2022}, editor = {Johansson, Ulf and Boström, Henrik and An Nguyen, Khuong and Luo, Zhiyuan and Carlsson, Lars}, volume = {179}, series = {Proceedings of Machine Learning Research}, month = {24--26 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v179/kirstein22a/kirstein22a.pdf}, url = {https://proceedings.mlr.press/v179/kirstein22a.html}, abstract = {We propose a new kernel learning approach based on efficient low-rank tensor compression for Gaussian process (GP) regression. The central idea is to compose a low-rank function represented in a hierarchical tensor format with a GP covariance function. Compared to similar deep neural network architectures, this approach facilitates to learn significantly more expressive features at lower computational costs as illustrated in the examples. Additionally, over-fitting is avoided with this compositional model by taking advantage of its inherent regularisation properties. Estimates of the generalisation error are compared to five baseline models on three synthetic and six real-world data sets. The experimental results show that the incorporated tensor network enables a highly accurate GP regression with a comparatively low number of trainable parameters. The observed performance is clearly superior (usually by an order of magnitude in mean squared error) to all examined standard models, in particular to deep neural networks with more than 1000 times as many parameters. } }
Endnote
%0 Conference Paper %T Tensor-Train Kernel Learning for Gaussian Processes %A Max Kirstein %A David Sommer %A Martin Eigel %B Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2022 %E Ulf Johansson %E Henrik Boström %E Khuong An Nguyen %E Zhiyuan Luo %E Lars Carlsson %F pmlr-v179-kirstein22a %I PMLR %P 253--272 %U https://proceedings.mlr.press/v179/kirstein22a.html %V 179 %X We propose a new kernel learning approach based on efficient low-rank tensor compression for Gaussian process (GP) regression. The central idea is to compose a low-rank function represented in a hierarchical tensor format with a GP covariance function. Compared to similar deep neural network architectures, this approach facilitates to learn significantly more expressive features at lower computational costs as illustrated in the examples. Additionally, over-fitting is avoided with this compositional model by taking advantage of its inherent regularisation properties. Estimates of the generalisation error are compared to five baseline models on three synthetic and six real-world data sets. The experimental results show that the incorporated tensor network enables a highly accurate GP regression with a comparatively low number of trainable parameters. The observed performance is clearly superior (usually by an order of magnitude in mean squared error) to all examined standard models, in particular to deep neural networks with more than 1000 times as many parameters.
APA
Kirstein, M., Sommer, D. & Eigel, M.. (2022). Tensor-Train Kernel Learning for Gaussian Processes. Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 179:253-272 Available from https://proceedings.mlr.press/v179/kirstein22a.html.

Related Material