On the Expected Size of Conformal Prediction Sets

Guneet S. Dhillon, George Deligiannidis, Tom Rainforth
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:1549-1557, 2024.

Abstract

While conformal predictors reap the benefits of rigorous statistical guarantees on their error frequency, the size of their corresponding prediction sets is critical to their practical utility. Unfortunately, there is currently a lack of finite-sample analysis and guarantees for their prediction set sizes. To address this shortfall, we theoretically quantify the expected size of the prediction sets under the split conformal prediction framework. As this precise formulation cannot usually be calculated directly, we further derive point estimates and high-probability interval bounds that can be empirically computed, providing a practical method for characterizing the expected set size. We corroborate the efficacy of our results with experiments on real-world datasets for both regression and classification problems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v238-dhillon24a, title = {On the Expected Size of Conformal Prediction Sets}, author = {Dhillon, Guneet S. and Deligiannidis, George and Rainforth, Tom}, booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics}, pages = {1549--1557}, year = {2024}, editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen}, volume = {238}, series = {Proceedings of Machine Learning Research}, month = {02--04 May}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v238/dhillon24a/dhillon24a.pdf}, url = {https://proceedings.mlr.press/v238/dhillon24a.html}, abstract = {While conformal predictors reap the benefits of rigorous statistical guarantees on their error frequency, the size of their corresponding prediction sets is critical to their practical utility. Unfortunately, there is currently a lack of finite-sample analysis and guarantees for their prediction set sizes. To address this shortfall, we theoretically quantify the expected size of the prediction sets under the split conformal prediction framework. As this precise formulation cannot usually be calculated directly, we further derive point estimates and high-probability interval bounds that can be empirically computed, providing a practical method for characterizing the expected set size. We corroborate the efficacy of our results with experiments on real-world datasets for both regression and classification problems.} }
Endnote
%0 Conference Paper %T On the Expected Size of Conformal Prediction Sets %A Guneet S. Dhillon %A George Deligiannidis %A Tom Rainforth %B Proceedings of The 27th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2024 %E Sanjoy Dasgupta %E Stephan Mandt %E Yingzhen Li %F pmlr-v238-dhillon24a %I PMLR %P 1549--1557 %U https://proceedings.mlr.press/v238/dhillon24a.html %V 238 %X While conformal predictors reap the benefits of rigorous statistical guarantees on their error frequency, the size of their corresponding prediction sets is critical to their practical utility. Unfortunately, there is currently a lack of finite-sample analysis and guarantees for their prediction set sizes. To address this shortfall, we theoretically quantify the expected size of the prediction sets under the split conformal prediction framework. As this precise formulation cannot usually be calculated directly, we further derive point estimates and high-probability interval bounds that can be empirically computed, providing a practical method for characterizing the expected set size. We corroborate the efficacy of our results with experiments on real-world datasets for both regression and classification problems.
APA
Dhillon, G.S., Deligiannidis, G. & Rainforth, T.. (2024). On the Expected Size of Conformal Prediction Sets. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 238:1549-1557 Available from https://proceedings.mlr.press/v238/dhillon24a.html.

Related Material