Exploring the Link Between Out-of-Distribution Detection and Conformal Prediction with Illustrations of Its Benefits

Paul Novello, Joseba Dalmau, Leo Andeol
Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 329:887-915, 2026.

Abstract

Research on Out-Of-Distribution (OOD) detection focuses mainly on building scores that efficiently distinguish OOD data from In Distribution (ID) data. On the other hand, Conformal Prediction (CP) uses non-conformity scores to construct prediction sets with probabilistic coverage guarantees. In other words, the former designs scores, while the latter designs probabilistic guarantees based on scores. In this paper, we study how these two fields can benefit each other, and we formalize several aspects of this connection. First, for OOD detection, we show that in standard OOD benchmark settings, evaluation metrics can be affected by the validation dataset’s finite sample size. Extending the work of Bates et al. (2023), we define new conformal AUROC and conformal FPR@TPR95 metrics, which are corrections that provide probabilistic guarantees on the variability of the FPR involved in these metrics with respect to the validation datasets. We show the effect of these corrections on two reference OOD and anomaly detection benchmarks, OpenOOD (Yang et al., 2022), and ADBench (Han et al., 2022). Second, for CP, we study the use of OOD scores as non-conformity scores and show that they can improve the efficiency of the prediction sets obtained with CP in several settings.

Cite this Paper

BibTeX
@InProceedings{pmlr-v329-novello26a, title = {Exploring the Link Between Out-of-Distribution Detection and Conformal Prediction with Illustrations of Its Benefits}, author = {Novello, Paul and Dalmau, Joseba and Andeol, Leo}, booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {887--915}, year = {2026}, editor = {Ahlberg, Ernst and Johansson, Ulf and Boström, Henrik and Carlevaro, Alberto and Hallberg Szabadváry, Johan and Carlsson, Lars}, volume = {329}, series = {Proceedings of Machine Learning Research}, month = {02--04 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v329/main/assets/novello26a/novello26a.pdf}, url = {https://proceedings.mlr.press/v329/novello26a.html}, abstract = {Research on Out-Of-Distribution (OOD) detection focuses mainly on building scores that efficiently distinguish OOD data from In Distribution (ID) data. On the other hand, Conformal Prediction (CP) uses non-conformity scores to construct prediction sets with probabilistic coverage guarantees. In other words, the former designs scores, while the latter designs probabilistic guarantees based on scores. In this paper, we study how these two fields can benefit each other, and we formalize several aspects of this connection. First, for OOD detection, we show that in standard OOD benchmark settings, evaluation metrics can be affected by the validation dataset’s finite sample size. Extending the work of Bates et al. (2023), we define new conformal AUROC and conformal FPR@TPR95 metrics, which are corrections that provide probabilistic guarantees on the variability of the FPR involved in these metrics with respect to the validation datasets. We show the effect of these corrections on two reference OOD and anomaly detection benchmarks, OpenOOD (Yang et al., 2022), and ADBench (Han et al., 2022). Second, for CP, we study the use of OOD scores as non-conformity scores and show that they can improve the efficiency of the prediction sets obtained with CP in several settings.} }
Endnote
%0 Conference Paper %T Exploring the Link Between Out-of-Distribution Detection and Conformal Prediction with Illustrations of Its Benefits %A Paul Novello %A Joseba Dalmau %A Leo Andeol %B Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2026 %E Ernst Ahlberg %E Ulf Johansson %E Henrik Boström %E Alberto Carlevaro %E Johan Hallberg Szabadváry %E Lars Carlsson %F pmlr-v329-novello26a %I PMLR %P 887--915 %U https://proceedings.mlr.press/v329/novello26a.html %V 329 %X Research on Out-Of-Distribution (OOD) detection focuses mainly on building scores that efficiently distinguish OOD data from In Distribution (ID) data. On the other hand, Conformal Prediction (CP) uses non-conformity scores to construct prediction sets with probabilistic coverage guarantees. In other words, the former designs scores, while the latter designs probabilistic guarantees based on scores. In this paper, we study how these two fields can benefit each other, and we formalize several aspects of this connection. First, for OOD detection, we show that in standard OOD benchmark settings, evaluation metrics can be affected by the validation dataset’s finite sample size. Extending the work of Bates et al. (2023), we define new conformal AUROC and conformal FPR@TPR95 metrics, which are corrections that provide probabilistic guarantees on the variability of the FPR involved in these metrics with respect to the validation datasets. We show the effect of these corrections on two reference OOD and anomaly detection benchmarks, OpenOOD (Yang et al., 2022), and ADBench (Han et al., 2022). Second, for CP, we study the use of OOD scores as non-conformity scores and show that they can improve the efficiency of the prediction sets obtained with CP in several settings.
APA
Novello, P., Dalmau, J. & Andeol, L.. (2026). Exploring the Link Between Out-of-Distribution Detection and Conformal Prediction with Illustrations of Its Benefits. Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 329:887-915 Available from https://proceedings.mlr.press/v329/novello26a.html.

Related Material