Conformal Prediction as Bayesian Quadrature

Jake C. Snell, Thomas L. Griffiths
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:56068-56084, 2025.

Abstract

As machine learning-based prediction systems are increasingly used in high-stakes situations, it is important to understand how such predictive models will perform upon deployment. Distribution-free uncertainty quantification techniques such as conformal prediction provide guarantees about the loss black-box models will incur even when the details of the models are hidden. However, such methods are based on frequentist probability, which unduly limits their applicability. We revisit the central aspects of conformal prediction from a Bayesian perspective and thereby illuminate the shortcomings of frequentist guarantees. We propose a practical alternative based on Bayesian quadrature that provides interpretable guarantees and offers a richer representation of the likely range of losses to be observed at test time.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-snell25a, title = {Conformal Prediction as {B}ayesian Quadrature}, author = {Snell, Jake C. and Griffiths, Thomas L.}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {56068--56084}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/snell25a/snell25a.pdf}, url = {https://proceedings.mlr.press/v267/snell25a.html}, abstract = {As machine learning-based prediction systems are increasingly used in high-stakes situations, it is important to understand how such predictive models will perform upon deployment. Distribution-free uncertainty quantification techniques such as conformal prediction provide guarantees about the loss black-box models will incur even when the details of the models are hidden. However, such methods are based on frequentist probability, which unduly limits their applicability. We revisit the central aspects of conformal prediction from a Bayesian perspective and thereby illuminate the shortcomings of frequentist guarantees. We propose a practical alternative based on Bayesian quadrature that provides interpretable guarantees and offers a richer representation of the likely range of losses to be observed at test time.} }
Endnote
%0 Conference Paper %T Conformal Prediction as Bayesian Quadrature %A Jake C. Snell %A Thomas L. Griffiths %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-snell25a %I PMLR %P 56068--56084 %U https://proceedings.mlr.press/v267/snell25a.html %V 267 %X As machine learning-based prediction systems are increasingly used in high-stakes situations, it is important to understand how such predictive models will perform upon deployment. Distribution-free uncertainty quantification techniques such as conformal prediction provide guarantees about the loss black-box models will incur even when the details of the models are hidden. However, such methods are based on frequentist probability, which unduly limits their applicability. We revisit the central aspects of conformal prediction from a Bayesian perspective and thereby illuminate the shortcomings of frequentist guarantees. We propose a practical alternative based on Bayesian quadrature that provides interpretable guarantees and offers a richer representation of the likely range of losses to be observed at test time.
APA
Snell, J.C. & Griffiths, T.L.. (2025). Conformal Prediction as Bayesian Quadrature. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:56068-56084 Available from https://proceedings.mlr.press/v267/snell25a.html.

Related Material