Questioning the Coverage-Length Metric in Conformal Prediction: When Shorter Intervals Are Not Better

Yizhou Min, Yizhou Lu, Lanqi Li, Zhen Zhang, Jiaye Teng
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:88935-88963, 2026.

Abstract

Conformal prediction (CP) has become a cornerstone of distribution-free uncertainty quantification, conventionally evaluated by its coverage and interval length. This work critically examines the sufficiency of these standard metrics. We demonstrate that the interval length might be deceptively improved through a counter-intuitive approach termed Prejudicial Trick (PT), while the coverage remains valid. Specifically, for any given test sample, PT probabilistically returns an interval, which is either null or constructed using an adjusted confidence level, thereby preserving marginal coverage. While PT potentially yields a deceptively lower interval length, it introduces practical vulnerabilities: the same input can yield completely different prediction intervals across repeated runs of the algorithm. We formally derive the conditions under which PT achieves these misleading improvements and provide extensive empirical evidence across various regression and classification tasks. Furthermore, we introduce a new metric interval stability which helps detect whether a new CP method implicitly improves the length based on such PT-like techniques. Code is available at https://github.com/benben-cd/PT-Conformal-Prediction

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-min26c, title = {Questioning the Coverage-Length Metric in Conformal Prediction: When Shorter Intervals Are Not Better}, author = {Min, Yizhou and Lu, Yizhou and Li, Lanqi and Zhang, Zhen and Teng, Jiaye}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {88935--88963}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/min26c/min26c.pdf}, url = {https://proceedings.mlr.press/v306/min26c.html}, abstract = {Conformal prediction (CP) has become a cornerstone of distribution-free uncertainty quantification, conventionally evaluated by its coverage and interval length. This work critically examines the sufficiency of these standard metrics. We demonstrate that the interval length might be deceptively improved through a counter-intuitive approach termed Prejudicial Trick (PT), while the coverage remains valid. Specifically, for any given test sample, PT probabilistically returns an interval, which is either null or constructed using an adjusted confidence level, thereby preserving marginal coverage. While PT potentially yields a deceptively lower interval length, it introduces practical vulnerabilities: the same input can yield completely different prediction intervals across repeated runs of the algorithm. We formally derive the conditions under which PT achieves these misleading improvements and provide extensive empirical evidence across various regression and classification tasks. Furthermore, we introduce a new metric interval stability which helps detect whether a new CP method implicitly improves the length based on such PT-like techniques. Code is available at https://github.com/benben-cd/PT-Conformal-Prediction} }
Endnote
%0 Conference Paper %T Questioning the Coverage-Length Metric in Conformal Prediction: When Shorter Intervals Are Not Better %A Yizhou Min %A Yizhou Lu %A Lanqi Li %A Zhen Zhang %A Jiaye Teng %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-min26c %I PMLR %P 88935--88963 %U https://proceedings.mlr.press/v306/min26c.html %V 306 %X Conformal prediction (CP) has become a cornerstone of distribution-free uncertainty quantification, conventionally evaluated by its coverage and interval length. This work critically examines the sufficiency of these standard metrics. We demonstrate that the interval length might be deceptively improved through a counter-intuitive approach termed Prejudicial Trick (PT), while the coverage remains valid. Specifically, for any given test sample, PT probabilistically returns an interval, which is either null or constructed using an adjusted confidence level, thereby preserving marginal coverage. While PT potentially yields a deceptively lower interval length, it introduces practical vulnerabilities: the same input can yield completely different prediction intervals across repeated runs of the algorithm. We formally derive the conditions under which PT achieves these misleading improvements and provide extensive empirical evidence across various regression and classification tasks. Furthermore, we introduce a new metric interval stability which helps detect whether a new CP method implicitly improves the length based on such PT-like techniques. Code is available at https://github.com/benben-cd/PT-Conformal-Prediction
APA
Min, Y., Lu, Y., Li, L., Zhang, Z. & Teng, J.. (2026). Questioning the Coverage-Length Metric in Conformal Prediction: When Shorter Intervals Are Not Better. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:88935-88963 Available from https://proceedings.mlr.press/v306/min26c.html.

Related Material