Spherical Inducing Features for Orthogonally-Decoupled Gaussian Processes

Louis C. Tiao, Vincent Dutordoir, Victor Picheny
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:34143-34160, 2023.

Abstract

Despite their many desirable properties, Gaussian processes (GPs) are often compared unfavorably to deep neural networks (NNs) for lacking the ability to learn representations. Recent efforts to bridge the gap between GPs and deep NNs have yielded a new class of inter-domain variational GPs in which the inducing variables correspond to hidden units of a feedforward NN. In this work, we examine some practical issues associated with this approach and propose an extension that leverages the orthogonal decomposition of GPs to mitigate these limitations. In particular, we introduce spherical inter-domain features to construct more flexible data-dependent basis functions for both the principal and orthogonal components of the GP approximation and show that incorporating NN activation features under this framework not only alleviates these shortcomings but is more scalable than alternative strategies. Experiments on multiple benchmark datasets demonstrate the effectiveness of our approach.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-tiao23a, title = {Spherical Inducing Features for Orthogonally-Decoupled {G}aussian Processes}, author = {Tiao, Louis C. and Dutordoir, Vincent and Picheny, Victor}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {34143--34160}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/tiao23a/tiao23a.pdf}, url = {https://proceedings.mlr.press/v202/tiao23a.html}, abstract = {Despite their many desirable properties, Gaussian processes (GPs) are often compared unfavorably to deep neural networks (NNs) for lacking the ability to learn representations. Recent efforts to bridge the gap between GPs and deep NNs have yielded a new class of inter-domain variational GPs in which the inducing variables correspond to hidden units of a feedforward NN. In this work, we examine some practical issues associated with this approach and propose an extension that leverages the orthogonal decomposition of GPs to mitigate these limitations. In particular, we introduce spherical inter-domain features to construct more flexible data-dependent basis functions for both the principal and orthogonal components of the GP approximation and show that incorporating NN activation features under this framework not only alleviates these shortcomings but is more scalable than alternative strategies. Experiments on multiple benchmark datasets demonstrate the effectiveness of our approach.} }
Endnote
%0 Conference Paper %T Spherical Inducing Features for Orthogonally-Decoupled Gaussian Processes %A Louis C. Tiao %A Vincent Dutordoir %A Victor Picheny %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-tiao23a %I PMLR %P 34143--34160 %U https://proceedings.mlr.press/v202/tiao23a.html %V 202 %X Despite their many desirable properties, Gaussian processes (GPs) are often compared unfavorably to deep neural networks (NNs) for lacking the ability to learn representations. Recent efforts to bridge the gap between GPs and deep NNs have yielded a new class of inter-domain variational GPs in which the inducing variables correspond to hidden units of a feedforward NN. In this work, we examine some practical issues associated with this approach and propose an extension that leverages the orthogonal decomposition of GPs to mitigate these limitations. In particular, we introduce spherical inter-domain features to construct more flexible data-dependent basis functions for both the principal and orthogonal components of the GP approximation and show that incorporating NN activation features under this framework not only alleviates these shortcomings but is more scalable than alternative strategies. Experiments on multiple benchmark datasets demonstrate the effectiveness of our approach.
APA
Tiao, L.C., Dutordoir, V. & Picheny, V.. (2023). Spherical Inducing Features for Orthogonally-Decoupled Gaussian Processes. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:34143-34160 Available from https://proceedings.mlr.press/v202/tiao23a.html.

Related Material