Multi-class Classification with Reject Option and Performance Guarantees using Conformal Prediction

Alberto García-Galindo, Marcos López-De-Castro, Rubén Armañanzas
Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 230:295-314, 2024.

Abstract

Beyond the standard classification scenario, allowing a classifier to refrain from making a prediction under uncertainty can have advantages in safety-critical applications, where a mistake may hold great costs. In this paper, we extend previous works on the development of classifiers with reject option grounded on the conformal prediction framework. Specifically, our work introduces a novel approach for inducing multi-class classifiers with reliable accuracy or recall estimates for a given rejection rate. We empirically evaluate our suggested approach in six multi-class datasets and demonstrate its effectiveness against both calibrated and uncalibrated probabilistic classifiers. The results underscore our method’s capability to provide reliable error rate estimates, thereby enhancing decision-making processes where erroneous predictions bear critical consequences.

Cite this Paper

BibTeX
@InProceedings{pmlr-v230-garcia-galindo24a, title = {Multi-class Classification with Reject Option and Performance Guarantees using Conformal Prediction}, author = {Garc\'ia-Galindo, Alberto and L\'opez-De-Castro, Marcos and Arma\~nanzas, Rub\'en}, booktitle = {Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {295--314}, year = {2024}, editor = {Vantini, Simone and Fontana, Matteo and Solari, Aldo and Boström, Henrik and Carlsson, Lars}, volume = {230}, series = {Proceedings of Machine Learning Research}, month = {09--11 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v230/main/assets/garcia-galindo24a/garcia-galindo24a.pdf}, url = {https://proceedings.mlr.press/v230/garcia-galindo24a.html}, abstract = {Beyond the standard classification scenario, allowing a classifier to refrain from making a prediction under uncertainty can have advantages in safety-critical applications, where a mistake may hold great costs. In this paper, we extend previous works on the development of classifiers with reject option grounded on the conformal prediction framework. Specifically, our work introduces a novel approach for inducing multi-class classifiers with reliable accuracy or recall estimates for a given rejection rate. We empirically evaluate our suggested approach in six multi-class datasets and demonstrate its effectiveness against both calibrated and uncalibrated probabilistic classifiers. The results underscore our method’s capability to provide reliable error rate estimates, thereby enhancing decision-making processes where erroneous predictions bear critical consequences.} }
Endnote
%0 Conference Paper %T Multi-class Classification with Reject Option and Performance Guarantees using Conformal Prediction %A Alberto García-Galindo %A Marcos López-De-Castro %A Rubén Armañanzas %B Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2024 %E Simone Vantini %E Matteo Fontana %E Aldo Solari %E Henrik Boström %E Lars Carlsson %F pmlr-v230-garcia-galindo24a %I PMLR %P 295--314 %U https://proceedings.mlr.press/v230/garcia-galindo24a.html %V 230 %X Beyond the standard classification scenario, allowing a classifier to refrain from making a prediction under uncertainty can have advantages in safety-critical applications, where a mistake may hold great costs. In this paper, we extend previous works on the development of classifiers with reject option grounded on the conformal prediction framework. Specifically, our work introduces a novel approach for inducing multi-class classifiers with reliable accuracy or recall estimates for a given rejection rate. We empirically evaluate our suggested approach in six multi-class datasets and demonstrate its effectiveness against both calibrated and uncalibrated probabilistic classifiers. The results underscore our method’s capability to provide reliable error rate estimates, thereby enhancing decision-making processes where erroneous predictions bear critical consequences.
APA
García-Galindo, A., López-De-Castro, M. & Armañanzas, R.. (2024). Multi-class Classification with Reject Option and Performance Guarantees using Conformal Prediction. Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 230:295-314 Available from https://proceedings.mlr.press/v230/garcia-galindo24a.html.

Related Material