Optimal Conformal Prediction under Epistemic Uncertainty

Alireza Javanmardi, Soroush H. Zargarbashi, Santo M. A. R. Thies, Willem Waegeman, Aleksandar Bojchevski, Eyke Hüllermeier
Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337:2461-2479, 2026.

Abstract

Conformal prediction ({CP}) is a widely used frequentist framework to quantify uncertainty by constructing prediction sets with user-specified marginal coverage guarantees. In practice, {CP} is typically applied on top of probabilistic classifiers, which are able to express aleatoric but not epistemic uncertainty. In this paper, we consider the question of how to optimally employ {CP} on top of a more expressive formalism, namely credal sets, which can express both aleatoric and epistemic uncertainty. More specifically, we propose probabilistic {Bernoulli} prediction sets and derive a variant that achieves conditional coverage for valid credal sets while remaining minimal in expected size. We then address the more realistic scenario in which the validity of the credal sets is not guaranteed. Assuming access to calibration data with ground-truth distributions over labels, we apply conformal risk control to BPS and derive a {PAC}-style guarantee: with high probability over the data, the achieved conditional coverage is at least the desired level. We validate our theoretical findings empirically over various datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v337-javanmardi26a, title = {Optimal Conformal Prediction under Epistemic Uncertainty}, author = {Javanmardi, Alireza and H. Zargarbashi, Soroush and Thies, Santo M. A. R. and Waegeman, Willem and Bojchevski, Aleksandar and H\"{u}llermeier, Eyke}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {2461--2479}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/javanmardi26a/javanmardi26a.pdf}, url = {https://proceedings.mlr.press/v337/javanmardi26a.html}, abstract = {Conformal prediction ({CP}) is a widely used frequentist framework to quantify uncertainty by constructing prediction sets with user-specified marginal coverage guarantees. In practice, {CP} is typically applied on top of probabilistic classifiers, which are able to express aleatoric but not epistemic uncertainty. In this paper, we consider the question of how to optimally employ {CP} on top of a more expressive formalism, namely credal sets, which can express both aleatoric and epistemic uncertainty. More specifically, we propose probabilistic {Bernoulli} prediction sets and derive a variant that achieves conditional coverage for valid credal sets while remaining minimal in expected size. We then address the more realistic scenario in which the validity of the credal sets is not guaranteed. Assuming access to calibration data with ground-truth distributions over labels, we apply conformal risk control to BPS and derive a {PAC}-style guarantee: with high probability over the data, the achieved conditional coverage is at least the desired level. We validate our theoretical findings empirically over various datasets.} }
Endnote
%0 Conference Paper %T Optimal Conformal Prediction under Epistemic Uncertainty %A Alireza Javanmardi %A Soroush H. Zargarbashi %A Santo M. A. R. Thies %A Willem Waegeman %A Aleksandar Bojchevski %A Eyke Hüllermeier %B Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Emilija Perković %E Daniel Malinsky %F pmlr-v337-javanmardi26a %I PMLR %P 2461--2479 %U https://proceedings.mlr.press/v337/javanmardi26a.html %V 337 %X Conformal prediction ({CP}) is a widely used frequentist framework to quantify uncertainty by constructing prediction sets with user-specified marginal coverage guarantees. In practice, {CP} is typically applied on top of probabilistic classifiers, which are able to express aleatoric but not epistemic uncertainty. In this paper, we consider the question of how to optimally employ {CP} on top of a more expressive formalism, namely credal sets, which can express both aleatoric and epistemic uncertainty. More specifically, we propose probabilistic {Bernoulli} prediction sets and derive a variant that achieves conditional coverage for valid credal sets while remaining minimal in expected size. We then address the more realistic scenario in which the validity of the credal sets is not guaranteed. Assuming access to calibration data with ground-truth distributions over labels, we apply conformal risk control to BPS and derive a {PAC}-style guarantee: with high probability over the data, the achieved conditional coverage is at least the desired level. We validate our theoretical findings empirically over various datasets.
APA
Javanmardi, A., H. Zargarbashi, S., Thies, S.M.A.R., Waegeman, W., Bojchevski, A. & Hüllermeier, E.. (2026). Optimal Conformal Prediction under Epistemic Uncertainty. Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 337:2461-2479 Available from https://proceedings.mlr.press/v337/javanmardi26a.html.

Related Material