Learning calibrated belief functions from conformal predictions

Vitor Martin Bordini, Sébastien Destercke, Benjamin Quost
Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications, PMLR 215:311-320, 2023.

Abstract

We consider the problem of supervised classification. We focus on the problem of calibrating the classifier’s outputs. We show that the p-values provided by Inductive Conformal Prediction (ICP) can be interpreted as a possibility distribution over the set of classes. This allows us to use ICP to compute a predictive belief function which is calibrated by construction. We also propose a learning method which provides p-values in a simpler and faster way, by making use of a multi-output regression model. Results obtained on the Cifar10 and Digits data sets show that our approach is comparable to standard ICP in terms of accuracy and calibration, while offering a reduced complexity and avoiding the use of a calibration set.

Cite this Paper

BibTeX
@InProceedings{pmlr-v215-martin-bordini23a, title = {Learning calibrated belief functions from conformal predictions}, author = {Martin Bordini, Vitor and Destercke, S\'ebastien and Quost, Benjamin}, booktitle = {Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications}, pages = {311--320}, year = {2023}, editor = {Miranda, Enrique and Montes, Ignacio and Quaeghebeur, Erik and Vantaggi, Barbara}, volume = {215}, series = {Proceedings of Machine Learning Research}, month = {11--14 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v215/martin-bordini23a/martin-bordini23a.pdf}, url = {https://proceedings.mlr.press/v215/martin-bordini23a.html}, abstract = {We consider the problem of supervised classification. We focus on the problem of calibrating the classifier’s outputs. We show that the p-values provided by Inductive Conformal Prediction (ICP) can be interpreted as a possibility distribution over the set of classes. This allows us to use ICP to compute a predictive belief function which is calibrated by construction. We also propose a learning method which provides p-values in a simpler and faster way, by making use of a multi-output regression model. Results obtained on the Cifar10 and Digits data sets show that our approach is comparable to standard ICP in terms of accuracy and calibration, while offering a reduced complexity and avoiding the use of a calibration set.} }
Endnote
%0 Conference Paper %T Learning calibrated belief functions from conformal predictions %A Vitor Martin Bordini %A Sébastien Destercke %A Benjamin Quost %B Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications %C Proceedings of Machine Learning Research %D 2023 %E Enrique Miranda %E Ignacio Montes %E Erik Quaeghebeur %E Barbara Vantaggi %F pmlr-v215-martin-bordini23a %I PMLR %P 311--320 %U https://proceedings.mlr.press/v215/martin-bordini23a.html %V 215 %X We consider the problem of supervised classification. We focus on the problem of calibrating the classifier’s outputs. We show that the p-values provided by Inductive Conformal Prediction (ICP) can be interpreted as a possibility distribution over the set of classes. This allows us to use ICP to compute a predictive belief function which is calibrated by construction. We also propose a learning method which provides p-values in a simpler and faster way, by making use of a multi-output regression model. Results obtained on the Cifar10 and Digits data sets show that our approach is comparable to standard ICP in terms of accuracy and calibration, while offering a reduced complexity and avoiding the use of a calibration set.
APA
Martin Bordini, V., Destercke, S. & Quost, B.. (2023). Learning calibrated belief functions from conformal predictions. Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications, in Proceedings of Machine Learning Research 215:311-320 Available from https://proceedings.mlr.press/v215/martin-bordini23a.html.

Related Material