Risk-Averse Evaluation of Conformal Scores through Conditional Value-at-Risk

Sara Narteni, Alberto Carlevaro, Fabrizio Dabbene, Maurizio Mongelli
Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 329:463-485, 2026.

Abstract

Conformal prediction provides distribution-free guarantees by constructing prediction sets that contain the true outcome with a predefined probability, under the mild assumption of exchangeability. While this marginal coverage guarantee is powerful, it remains an aggregate measure, by controlling the frequency of errors but being agnostic to their severity. In many high-stakes applications, however, not all errors are equally critical. Two conformal predictors achieving the same nominal coverage may exhibit different tail behaviors: one may incur into large deviations from the empirical quantile, while another may keep closer to it. However, standard evaluation metrics, such as empirical coverage and average set size, fail to capture this aspect. To address this limitation, we propose to evaluate conformal predictors through the lens of Conditional Value-at-Risk, a well-established measure of tail risk. This perspective enables a risk-averse selection of score functions, favoring those that control not only the frequency but also the magnitude of errors. In particular, we select the score function that minimizes this tail risk, thereby ensuring a tighter alignment between the worst errors observed on test data and the quantile estimated during calibration.

Cite this Paper

BibTeX
@InProceedings{pmlr-v329-narteni26a, title = {Risk-Averse Evaluation of Conformal Scores through Conditional Value-at-Risk}, author = {Narteni, Sara and Carlevaro, Alberto and Dabbene, Fabrizio and Mongelli, Maurizio}, booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {463--485}, year = {2026}, editor = {Ahlberg, Ernst and Johansson, Ulf and Boström, Henrik and Carlevaro, Alberto and Hallberg Szabadváry, Johan and Carlsson, Lars}, volume = {329}, series = {Proceedings of Machine Learning Research}, month = {02--04 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v329/main/assets/narteni26a/narteni26a.pdf}, url = {https://proceedings.mlr.press/v329/narteni26a.html}, abstract = {Conformal prediction provides distribution-free guarantees by constructing prediction sets that contain the true outcome with a predefined probability, under the mild assumption of exchangeability. While this marginal coverage guarantee is powerful, it remains an aggregate measure, by controlling the frequency of errors but being agnostic to their severity. In many high-stakes applications, however, not all errors are equally critical. Two conformal predictors achieving the same nominal coverage may exhibit different tail behaviors: one may incur into large deviations from the empirical quantile, while another may keep closer to it. However, standard evaluation metrics, such as empirical coverage and average set size, fail to capture this aspect. To address this limitation, we propose to evaluate conformal predictors through the lens of Conditional Value-at-Risk, a well-established measure of tail risk. This perspective enables a risk-averse selection of score functions, favoring those that control not only the frequency but also the magnitude of errors. In particular, we select the score function that minimizes this tail risk, thereby ensuring a tighter alignment between the worst errors observed on test data and the quantile estimated during calibration.} }
Endnote
%0 Conference Paper %T Risk-Averse Evaluation of Conformal Scores through Conditional Value-at-Risk %A Sara Narteni %A Alberto Carlevaro %A Fabrizio Dabbene %A Maurizio Mongelli %B Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2026 %E Ernst Ahlberg %E Ulf Johansson %E Henrik Boström %E Alberto Carlevaro %E Johan Hallberg Szabadváry %E Lars Carlsson %F pmlr-v329-narteni26a %I PMLR %P 463--485 %U https://proceedings.mlr.press/v329/narteni26a.html %V 329 %X Conformal prediction provides distribution-free guarantees by constructing prediction sets that contain the true outcome with a predefined probability, under the mild assumption of exchangeability. While this marginal coverage guarantee is powerful, it remains an aggregate measure, by controlling the frequency of errors but being agnostic to their severity. In many high-stakes applications, however, not all errors are equally critical. Two conformal predictors achieving the same nominal coverage may exhibit different tail behaviors: one may incur into large deviations from the empirical quantile, while another may keep closer to it. However, standard evaluation metrics, such as empirical coverage and average set size, fail to capture this aspect. To address this limitation, we propose to evaluate conformal predictors through the lens of Conditional Value-at-Risk, a well-established measure of tail risk. This perspective enables a risk-averse selection of score functions, favoring those that control not only the frequency but also the magnitude of errors. In particular, we select the score function that minimizes this tail risk, thereby ensuring a tighter alignment between the worst errors observed on test data and the quantile estimated during calibration.
APA
Narteni, S., Carlevaro, A., Dabbene, F. & Mongelli, M.. (2026). Risk-Averse Evaluation of Conformal Scores through Conditional Value-at-Risk. Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 329:463-485 Available from https://proceedings.mlr.press/v329/narteni26a.html.

Related Material