Well-calibrated probabilities for prediction sets

Oskar Gustafsson, John Pavia, Styrbjörn Käll, Johan Hallberg Szabadváry, Jens Lundström, Ernst Ahlberg
Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 329:222-248, 2026.

Abstract

Conformal prediction is a model-agnostic, distribution-free statistical framework that outputs prediction sets such that, for each prediction, the probability that the true label falls outside the set is less than a user-specified tolerance, $\varepsilon$. Notably, this guarantee holds conditionally on the calibration dataset and thus only prior to observing a new object. We present a method that assigns prediction probabilities to any set prediction. These probabilities approximate the true conditional probability that the label lies within the set, and converge to it with increasing data. Venn-Abers predictors are (multi)-probabilistic predictors combining Venn-predictors and isotonic regression to output well-calibrated probabilities for binary prediction problems. We propose applying Venn-Abers to the constructed prediction sets from conformal prediction and, thereby, assigning a well-calibrated probability of the true label being in that set. We further introduce a novel self-consistent scoring rule for Venn-Abers predictors, which treats a predicted probability as being as likely as the event it postulates.

Cite this Paper

BibTeX
@InProceedings{pmlr-v329-gustafsson26a, title = {Well-calibrated probabilities for prediction sets}, author = {Gustafsson, Oskar and Pavia, John and K{\"a}ll, Styrbj{\"o}rn and Szabadv{\'a}ry, Johan Hallberg and Lundstr{\"o}m, Jens and Ahlberg, Ernst}, booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {222--248}, year = {2026}, editor = {Ahlberg, Ernst and Johansson, Ulf and Boström, Henrik and Carlevaro, Alberto and Hallberg Szabadváry, Johan and Carlsson, Lars}, volume = {329}, series = {Proceedings of Machine Learning Research}, month = {02--04 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v329/main/assets/gustafsson26a/gustafsson26a.pdf}, url = {https://proceedings.mlr.press/v329/gustafsson26a.html}, abstract = {Conformal prediction is a model-agnostic, distribution-free statistical framework that outputs prediction sets such that, for each prediction, the probability that the true label falls outside the set is less than a user-specified tolerance, $\varepsilon$. Notably, this guarantee holds conditionally on the calibration dataset and thus only prior to observing a new object. We present a method that assigns prediction probabilities to any set prediction. These probabilities approximate the true conditional probability that the label lies within the set, and converge to it with increasing data. Venn-Abers predictors are (multi)-probabilistic predictors combining Venn-predictors and isotonic regression to output well-calibrated probabilities for binary prediction problems. We propose applying Venn-Abers to the constructed prediction sets from conformal prediction and, thereby, assigning a well-calibrated probability of the true label being in that set. We further introduce a novel self-consistent scoring rule for Venn-Abers predictors, which treats a predicted probability as being as likely as the event it postulates.} }
Endnote
%0 Conference Paper %T Well-calibrated probabilities for prediction sets %A Oskar Gustafsson %A John Pavia %A Styrbjörn Käll %A Johan Hallberg Szabadváry %A Jens Lundström %A Ernst Ahlberg %B Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2026 %E Ernst Ahlberg %E Ulf Johansson %E Henrik Boström %E Alberto Carlevaro %E Johan Hallberg Szabadváry %E Lars Carlsson %F pmlr-v329-gustafsson26a %I PMLR %P 222--248 %U https://proceedings.mlr.press/v329/gustafsson26a.html %V 329 %X Conformal prediction is a model-agnostic, distribution-free statistical framework that outputs prediction sets such that, for each prediction, the probability that the true label falls outside the set is less than a user-specified tolerance, $\varepsilon$. Notably, this guarantee holds conditionally on the calibration dataset and thus only prior to observing a new object. We present a method that assigns prediction probabilities to any set prediction. These probabilities approximate the true conditional probability that the label lies within the set, and converge to it with increasing data. Venn-Abers predictors are (multi)-probabilistic predictors combining Venn-predictors and isotonic regression to output well-calibrated probabilities for binary prediction problems. We propose applying Venn-Abers to the constructed prediction sets from conformal prediction and, thereby, assigning a well-calibrated probability of the true label being in that set. We further introduce a novel self-consistent scoring rule for Venn-Abers predictors, which treats a predicted probability as being as likely as the event it postulates.
APA
Gustafsson, O., Pavia, J., Käll, S., Szabadváry, J.H., Lundström, J. & Ahlberg, E.. (2026). Well-calibrated probabilities for prediction sets. Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 329:222-248 Available from https://proceedings.mlr.press/v329/gustafsson26a.html.

Related Material