Impact of model-agnostic nonconformity functions on efficiency of conformal classifiers: an extensive study

Marharyta Aleksandrova, Oleg Chertov
Proceedings of the Tenth Symposium on Conformal and Probabilistic Prediction and Applications, PMLR 152:151-170, 2021.

Abstract

The property of conformal predictors to guarantee the required accuracy rate makes this framework attractive in various practical applications. However, this property is achieved at a price of reduction in precision. In the case of conformal classification, the system can output multiple class labels instead of one. It is also known, that the choice of nonconformity function has a major impact on the efficiency of conformal classifiers. Recently, it was shown that different model-agnostic nonconformity functions result in conformal classifiers with different characteristics. For a Neural Network-based conformal classifier, the \emph{inverse probability} (or hinge loss) allows minimizing the average number of predicted labels, and \emph{margin} results in a larger fraction of singleton predictions. In this work, we aim to further extend this study. We perform an experimental evaluation using 8 different classification algorithms and discuss when the previously observed relationship holds or not. Additionally, we propose a successful method to combine the properties of these two nonconformity functions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v152-aleksandrova21a, title = {Impact of model-agnostic nonconformity functions on efficiency of conformal classifiers: an extensive study}, author = {Aleksandrova, Marharyta and Chertov, Oleg}, booktitle = {Proceedings of the Tenth Symposium on Conformal and Probabilistic Prediction and Applications}, pages = {151--170}, year = {2021}, editor = {Carlsson, Lars and Luo, Zhiyuan and Cherubin, Giovanni and An Nguyen, Khuong}, volume = {152}, series = {Proceedings of Machine Learning Research}, month = {08--10 Sep}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v152/aleksandrova21a/aleksandrova21a.pdf}, url = {https://proceedings.mlr.press/v152/aleksandrova21a.html}, abstract = {The property of conformal predictors to guarantee the required accuracy rate makes this framework attractive in various practical applications. However, this property is achieved at a price of reduction in precision. In the case of conformal classification, the system can output multiple class labels instead of one. It is also known, that the choice of nonconformity function has a major impact on the efficiency of conformal classifiers. Recently, it was shown that different model-agnostic nonconformity functions result in conformal classifiers with different characteristics. For a Neural Network-based conformal classifier, the \emph{inverse probability} (or hinge loss) allows minimizing the average number of predicted labels, and \emph{margin} results in a larger fraction of singleton predictions. In this work, we aim to further extend this study. We perform an experimental evaluation using 8 different classification algorithms and discuss when the previously observed relationship holds or not. Additionally, we propose a successful method to combine the properties of these two nonconformity functions.} }
Endnote
%0 Conference Paper %T Impact of model-agnostic nonconformity functions on efficiency of conformal classifiers: an extensive study %A Marharyta Aleksandrova %A Oleg Chertov %B Proceedings of the Tenth Symposium on Conformal and Probabilistic Prediction and Applications %C Proceedings of Machine Learning Research %D 2021 %E Lars Carlsson %E Zhiyuan Luo %E Giovanni Cherubin %E Khuong An Nguyen %F pmlr-v152-aleksandrova21a %I PMLR %P 151--170 %U https://proceedings.mlr.press/v152/aleksandrova21a.html %V 152 %X The property of conformal predictors to guarantee the required accuracy rate makes this framework attractive in various practical applications. However, this property is achieved at a price of reduction in precision. In the case of conformal classification, the system can output multiple class labels instead of one. It is also known, that the choice of nonconformity function has a major impact on the efficiency of conformal classifiers. Recently, it was shown that different model-agnostic nonconformity functions result in conformal classifiers with different characteristics. For a Neural Network-based conformal classifier, the \emph{inverse probability} (or hinge loss) allows minimizing the average number of predicted labels, and \emph{margin} results in a larger fraction of singleton predictions. In this work, we aim to further extend this study. We perform an experimental evaluation using 8 different classification algorithms and discuss when the previously observed relationship holds or not. Additionally, we propose a successful method to combine the properties of these two nonconformity functions.
APA
Aleksandrova, M. & Chertov, O.. (2021). Impact of model-agnostic nonconformity functions on efficiency of conformal classifiers: an extensive study. Proceedings of the Tenth Symposium on Conformal and Probabilistic Prediction and Applications, in Proceedings of Machine Learning Research 152:151-170 Available from https://proceedings.mlr.press/v152/aleksandrova21a.html.

Related Material