Evidential Uncertainty Sets in Deep Classifiers Using Conformal Prediction

Hamed Karimi, Reza Samavi
Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 230:466-489, 2024.

Abstract

In this paper, we propose \emph{Evidential Conformal Prediction (ECP)} method for deep classifiers to generate the conformal prediction sets. Our method is designed based on a non-conformity score function that has its roots in Evidential Deep Learning (EDL) as a method of quantifying model (epistemic) uncertainty in DNN classifiers. We use evidence that are derived from the logit values of target labels to compute the components of our non-conformity score function: the heuristic notion of uncertainty in CP, uncertainty surprisal, and expected utility. Our extensive experimental evaluation demonstrates that ECP outperforms three state-of-the-art methods for generating CP sets, in terms of their set sizes and adaptivity while maintaining the coverage of true labels.

Cite this Paper

BibTeX
@InProceedings{pmlr-v230-karimi24a, title = {Evidential Uncertainty Sets in Deep Classifiers Using Conformal Prediction}, author = {Karimi, Hamed and Samavi, Reza}, booktitle = {Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {466--489}, year = {2024}, editor = {Vantini, Simone and Fontana, Matteo and Solari, Aldo and Boström, Henrik and Carlsson, Lars}, volume = {230}, series = {Proceedings of Machine Learning Research}, month = {09--11 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v230/main/assets/karimi24a/karimi24a.pdf}, url = {https://proceedings.mlr.press/v230/karimi24a.html}, abstract = {In this paper, we propose \emph{Evidential Conformal Prediction (ECP)} method for deep classifiers to generate the conformal prediction sets. Our method is designed based on a non-conformity score function that has its roots in Evidential Deep Learning (EDL) as a method of quantifying model (epistemic) uncertainty in DNN classifiers. We use evidence that are derived from the logit values of target labels to compute the components of our non-conformity score function: the heuristic notion of uncertainty in CP, uncertainty surprisal, and expected utility. Our extensive experimental evaluation demonstrates that ECP outperforms three state-of-the-art methods for generating CP sets, in terms of their set sizes and adaptivity while maintaining the coverage of true labels.} }
Endnote
%0 Conference Paper %T Evidential Uncertainty Sets in Deep Classifiers Using Conformal Prediction %A Hamed Karimi %A Reza Samavi %B Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2024 %E Simone Vantini %E Matteo Fontana %E Aldo Solari %E Henrik Boström %E Lars Carlsson %F pmlr-v230-karimi24a %I PMLR %P 466--489 %U https://proceedings.mlr.press/v230/karimi24a.html %V 230 %X In this paper, we propose \emph{Evidential Conformal Prediction (ECP)} method for deep classifiers to generate the conformal prediction sets. Our method is designed based on a non-conformity score function that has its roots in Evidential Deep Learning (EDL) as a method of quantifying model (epistemic) uncertainty in DNN classifiers. We use evidence that are derived from the logit values of target labels to compute the components of our non-conformity score function: the heuristic notion of uncertainty in CP, uncertainty surprisal, and expected utility. Our extensive experimental evaluation demonstrates that ECP outperforms three state-of-the-art methods for generating CP sets, in terms of their set sizes and adaptivity while maintaining the coverage of true labels.
APA
Karimi, H. & Samavi, R.. (2024). Evidential Uncertainty Sets in Deep Classifiers Using Conformal Prediction. Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 230:466-489 Available from https://proceedings.mlr.press/v230/karimi24a.html.

Related Material