Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes

Liam Hodgkinson, Chris Van Der Heide, Fred Roosta, Michael W. Mahoney
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:13085-13117, 2023.

Abstract

Despite their importance for assessing reliability of predictions, uncertainty quantification (UQ) measures in machine learning models have only recently begun to be rigorously characterized. One prominent issue is the curse of dimensionality: it is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and both should deteriorate with larger input dimensions. However, we prove that by tuning hyperparameters to maximize marginal likelihood (the empirical Bayes procedure), performance, as measured by the marginal likelihood, improves monotonically with the input dimension. On the other hand, cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent. Cold posteriors, which have recently attracted interest due to their improved performance in certain settings, appear to exacerbate these phenomena. We verify empirically that our results hold for real data, beyond our considered assumptions, and we explore consequences involving synthetic covariates.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-hodgkinson23a, title = {Monotonicity and Double Descent in Uncertainty Estimation with {G}aussian Processes}, author = {Hodgkinson, Liam and Van Der Heide, Chris and Roosta, Fred and Mahoney, Michael W.}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {13085--13117}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/hodgkinson23a/hodgkinson23a.pdf}, url = {https://proceedings.mlr.press/v202/hodgkinson23a.html}, abstract = {Despite their importance for assessing reliability of predictions, uncertainty quantification (UQ) measures in machine learning models have only recently begun to be rigorously characterized. One prominent issue is the curse of dimensionality: it is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and both should deteriorate with larger input dimensions. However, we prove that by tuning hyperparameters to maximize marginal likelihood (the empirical Bayes procedure), performance, as measured by the marginal likelihood, improves monotonically with the input dimension. On the other hand, cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent. Cold posteriors, which have recently attracted interest due to their improved performance in certain settings, appear to exacerbate these phenomena. We verify empirically that our results hold for real data, beyond our considered assumptions, and we explore consequences involving synthetic covariates.} }
Endnote
%0 Conference Paper %T Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes %A Liam Hodgkinson %A Chris Van Der Heide %A Fred Roosta %A Michael W. Mahoney %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-hodgkinson23a %I PMLR %P 13085--13117 %U https://proceedings.mlr.press/v202/hodgkinson23a.html %V 202 %X Despite their importance for assessing reliability of predictions, uncertainty quantification (UQ) measures in machine learning models have only recently begun to be rigorously characterized. One prominent issue is the curse of dimensionality: it is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and both should deteriorate with larger input dimensions. However, we prove that by tuning hyperparameters to maximize marginal likelihood (the empirical Bayes procedure), performance, as measured by the marginal likelihood, improves monotonically with the input dimension. On the other hand, cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent. Cold posteriors, which have recently attracted interest due to their improved performance in certain settings, appear to exacerbate these phenomena. We verify empirically that our results hold for real data, beyond our considered assumptions, and we explore consequences involving synthetic covariates.
APA
Hodgkinson, L., Van Der Heide, C., Roosta, F. & Mahoney, M.W.. (2023). Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:13085-13117 Available from https://proceedings.mlr.press/v202/hodgkinson23a.html.

Related Material