Max-Rank: Efficient Multiple Testing for Conformal Prediction

Alexander Timans, Christoph-Nikolas Straehle, Kaspar Sakmann, Christian A. Naesseth, Eric Nalisnick
Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:3898-3906, 2025.

Abstract

Multiple hypothesis testing (MHT) frequently arises in scientific inquiries, and concurrent testing of multiple hypotheses inflates the risk of Type-I errors or false positives, rendering MHT corrections essential. This paper addresses MHT in the context of conformal prediction, a flexible framework for predictive uncertainty quantification. Some conformal applications give rise to simultaneous testing, and positive dependencies among tests typically exist. We introduce max-rank, a novel correction that exploits these dependencies whilst efficiently controlling the family-wise error rate. Inspired by existing permutation-based corrections, max-rank leverages rank order information to improve performance and integrates readily with any conformal procedure. We establish its theoretical and empirical advantages over the common Bonferroni correction and its compatibility with conformal prediction, highlighting the potential to strengthen predictive uncertainty estimates.

Cite this Paper

BibTeX
@InProceedings{pmlr-v258-timans25a, title = {Max-Rank: Efficient Multiple Testing for Conformal Prediction}, author = {Timans, Alexander and Straehle, Christoph-Nikolas and Sakmann, Kaspar and Naesseth, Christian A. and Nalisnick, Eric}, booktitle = {Proceedings of The 28th International Conference on Artificial Intelligence and Statistics}, pages = {3898--3906}, year = {2025}, editor = {Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz}, volume = {258}, series = {Proceedings of Machine Learning Research}, month = {03--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v258/main/assets/timans25a/timans25a.pdf}, url = {https://proceedings.mlr.press/v258/timans25a.html}, abstract = {Multiple hypothesis testing (MHT) frequently arises in scientific inquiries, and concurrent testing of multiple hypotheses inflates the risk of Type-I errors or false positives, rendering MHT corrections essential. This paper addresses MHT in the context of conformal prediction, a flexible framework for predictive uncertainty quantification. Some conformal applications give rise to simultaneous testing, and positive dependencies among tests typically exist. We introduce max-rank, a novel correction that exploits these dependencies whilst efficiently controlling the family-wise error rate. Inspired by existing permutation-based corrections, max-rank leverages rank order information to improve performance and integrates readily with any conformal procedure. We establish its theoretical and empirical advantages over the common Bonferroni correction and its compatibility with conformal prediction, highlighting the potential to strengthen predictive uncertainty estimates.} }
Endnote
%0 Conference Paper %T Max-Rank: Efficient Multiple Testing for Conformal Prediction %A Alexander Timans %A Christoph-Nikolas Straehle %A Kaspar Sakmann %A Christian A. Naesseth %A Eric Nalisnick %B Proceedings of The 28th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2025 %E Yingzhen Li %E Stephan Mandt %E Shipra Agrawal %E Emtiyaz Khan %F pmlr-v258-timans25a %I PMLR %P 3898--3906 %U https://proceedings.mlr.press/v258/timans25a.html %V 258 %X Multiple hypothesis testing (MHT) frequently arises in scientific inquiries, and concurrent testing of multiple hypotheses inflates the risk of Type-I errors or false positives, rendering MHT corrections essential. This paper addresses MHT in the context of conformal prediction, a flexible framework for predictive uncertainty quantification. Some conformal applications give rise to simultaneous testing, and positive dependencies among tests typically exist. We introduce max-rank, a novel correction that exploits these dependencies whilst efficiently controlling the family-wise error rate. Inspired by existing permutation-based corrections, max-rank leverages rank order information to improve performance and integrates readily with any conformal procedure. We establish its theoretical and empirical advantages over the common Bonferroni correction and its compatibility with conformal prediction, highlighting the potential to strengthen predictive uncertainty estimates.
APA
Timans, A., Straehle, C., Sakmann, K., Naesseth, C.A. & Nalisnick, E.. (2025). Max-Rank: Efficient Multiple Testing for Conformal Prediction. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 258:3898-3906 Available from https://proceedings.mlr.press/v258/timans25a.html.

Related Material