Conformal Regression with Reject Option

Ulf Johansson, Cecilia Sönströd, Henrik Boström
Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 230:277-294, 2024.

Abstract

A regressor with reject option may refrain from making predictions expected to be inaccurate. In this paper, we introduce and evaluate conformal regression with reject option. Consistent with standard conformal regression, non-rejected predictions are valid prediction intervals. The suggested approach utilizes Mondrian conformal regression, where the categories are dynamically created from difficulty estimations of individual instances and requested rejection levels. As shown in the experiments, using $16$ publicly available data sets and random forests as underlying models, the conformal regressors produced progressively tighter intervals for higher rejection levels, thus demonstrating the trade-off between coverage and informativeness targeted when adding a reject option. A key property of the novel method is the fact that the informativeness, i.e., the interval sizes, resulting from any combination of significance and rejection levels is known to the user before making any test predictions. While all four different difficulty estimators evaluated led to consistently tighter intervals for higher rejection levels, the one producing the most efficient conformal regressors utilized the disagreement between the trees in the Random forest.

Cite this Paper

BibTeX
@InProceedings{pmlr-v230-johansson24a, title = {Conformal Regression with Reject Option}, author = {Johansson, Ulf and S\"{o}nstr\"{o}d, Cecilia and Bostr\"{o}m, Henrik}, booktitle = {Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {277--294}, year = {2024}, editor = {Vantini, Simone and Fontana, Matteo and Solari, Aldo and Boström, Henrik and Carlsson, Lars}, volume = {230}, series = {Proceedings of Machine Learning Research}, month = {09--11 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v230/main/assets/johansson24a/johansson24a.pdf}, url = {https://proceedings.mlr.press/v230/johansson24a.html}, abstract = {A regressor with reject option may refrain from making predictions expected to be inaccurate. In this paper, we introduce and evaluate conformal regression with reject option. Consistent with standard conformal regression, non-rejected predictions are valid prediction intervals. The suggested approach utilizes Mondrian conformal regression, where the categories are dynamically created from difficulty estimations of individual instances and requested rejection levels. As shown in the experiments, using $16$ publicly available data sets and random forests as underlying models, the conformal regressors produced progressively tighter intervals for higher rejection levels, thus demonstrating the trade-off between coverage and informativeness targeted when adding a reject option. A key property of the novel method is the fact that the informativeness, i.e., the interval sizes, resulting from any combination of significance and rejection levels is known to the user before making any test predictions. While all four different difficulty estimators evaluated led to consistently tighter intervals for higher rejection levels, the one producing the most efficient conformal regressors utilized the disagreement between the trees in the Random forest.} }
Endnote
%0 Conference Paper %T Conformal Regression with Reject Option %A Ulf Johansson %A Cecilia Sönströd %A Henrik Boström %B Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2024 %E Simone Vantini %E Matteo Fontana %E Aldo Solari %E Henrik Boström %E Lars Carlsson %F pmlr-v230-johansson24a %I PMLR %P 277--294 %U https://proceedings.mlr.press/v230/johansson24a.html %V 230 %X A regressor with reject option may refrain from making predictions expected to be inaccurate. In this paper, we introduce and evaluate conformal regression with reject option. Consistent with standard conformal regression, non-rejected predictions are valid prediction intervals. The suggested approach utilizes Mondrian conformal regression, where the categories are dynamically created from difficulty estimations of individual instances and requested rejection levels. As shown in the experiments, using $16$ publicly available data sets and random forests as underlying models, the conformal regressors produced progressively tighter intervals for higher rejection levels, thus demonstrating the trade-off between coverage and informativeness targeted when adding a reject option. A key property of the novel method is the fact that the informativeness, i.e., the interval sizes, resulting from any combination of significance and rejection levels is known to the user before making any test predictions. While all four different difficulty estimators evaluated led to consistently tighter intervals for higher rejection levels, the one producing the most efficient conformal regressors utilized the disagreement between the trees in the Random forest.
APA
Johansson, U., Sönströd, C. & Boström, H.. (2024). Conformal Regression with Reject Option. Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 230:277-294 Available from https://proceedings.mlr.press/v230/johansson24a.html.

Related Material