Vecchia Gaussian Process Ensembles on Internal Representations of Deep Neural Networks

Felix Jimenez, Matthias Katzfuss
Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:3403-3411, 2025.

Abstract

For regression tasks, standard Gaussian processes (GPs) provide natural uncertainty quantification (UQ), while deep neural networks (DNNs) excel at representation learning. Deterministic UQ methods for neural networks have successfully combined the two and require only a single pass through the neural network. However, current methods necessitate changes to network training to address feature collapse, where unique inputs map to identical feature vectors. We propose an alternative solution, the deep Vecchia ensemble (DVE), which allows deterministic UQ to work in the presence of feature collapse, negating the need for network retraining. DVE comprises an ensemble of GPs built on hidden-layer outputs of a DNN, achieving scalability via Vecchia approximations that leverage nearest-neighbor conditional independence. DVE is compatible with pretrained networks and incurs low computational overhead. We demonstrate DVE’s utility on several datasets and carry out experiments to understand the inner workings of the proposed method.

Cite this Paper

BibTeX
@InProceedings{pmlr-v258-jimenez25a, title = {Vecchia Gaussian Process Ensembles on Internal Representations of Deep Neural Networks}, author = {Jimenez, Felix and Katzfuss, Matthias}, booktitle = {Proceedings of The 28th International Conference on Artificial Intelligence and Statistics}, pages = {3403--3411}, year = {2025}, editor = {Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz}, volume = {258}, series = {Proceedings of Machine Learning Research}, month = {03--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v258/main/assets/jimenez25a/jimenez25a.pdf}, url = {https://proceedings.mlr.press/v258/jimenez25a.html}, abstract = {For regression tasks, standard Gaussian processes (GPs) provide natural uncertainty quantification (UQ), while deep neural networks (DNNs) excel at representation learning. Deterministic UQ methods for neural networks have successfully combined the two and require only a single pass through the neural network. However, current methods necessitate changes to network training to address feature collapse, where unique inputs map to identical feature vectors. We propose an alternative solution, the deep Vecchia ensemble (DVE), which allows deterministic UQ to work in the presence of feature collapse, negating the need for network retraining. DVE comprises an ensemble of GPs built on hidden-layer outputs of a DNN, achieving scalability via Vecchia approximations that leverage nearest-neighbor conditional independence. DVE is compatible with pretrained networks and incurs low computational overhead. We demonstrate DVE’s utility on several datasets and carry out experiments to understand the inner workings of the proposed method.} }
Endnote
%0 Conference Paper %T Vecchia Gaussian Process Ensembles on Internal Representations of Deep Neural Networks %A Felix Jimenez %A Matthias Katzfuss %B Proceedings of The 28th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2025 %E Yingzhen Li %E Stephan Mandt %E Shipra Agrawal %E Emtiyaz Khan %F pmlr-v258-jimenez25a %I PMLR %P 3403--3411 %U https://proceedings.mlr.press/v258/jimenez25a.html %V 258 %X For regression tasks, standard Gaussian processes (GPs) provide natural uncertainty quantification (UQ), while deep neural networks (DNNs) excel at representation learning. Deterministic UQ methods for neural networks have successfully combined the two and require only a single pass through the neural network. However, current methods necessitate changes to network training to address feature collapse, where unique inputs map to identical feature vectors. We propose an alternative solution, the deep Vecchia ensemble (DVE), which allows deterministic UQ to work in the presence of feature collapse, negating the need for network retraining. DVE comprises an ensemble of GPs built on hidden-layer outputs of a DNN, achieving scalability via Vecchia approximations that leverage nearest-neighbor conditional independence. DVE is compatible with pretrained networks and incurs low computational overhead. We demonstrate DVE’s utility on several datasets and carry out experiments to understand the inner workings of the proposed method.
APA
Jimenez, F. & Katzfuss, M.. (2025). Vecchia Gaussian Process Ensembles on Internal Representations of Deep Neural Networks. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 258:3403-3411 Available from https://proceedings.mlr.press/v258/jimenez25a.html.

Related Material