Optimal Zero-shot Regret Minimization for Selective Classification with Out-of-Distribution Detection

Eduardo Dadalto Câmara Gomes, Marco Romanelli
Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence, PMLR 286:1497-1520, 2025.

Abstract

Selective Classification with Out-of-Distribution Detection (SCOD) is a general framework that combines the detection of incorrectly classified in-distribution samples and out-of-distribution samples. Previous solutions for SCOD heavily rely on the choice of Selective Classification (SC) and Out-of-Distribution (OOD) detectors selected at test time. Notably, the performance of these detectors varies across different underlying data distributions. Hence, a poor choice can affect the efficacy of the SCOD framework. On the other hand, making an informed choice is impossible without samples from both in- and out-distribution. We propose an optimal zero-shot black-box method for SCOD that aggregates off-the-shelf detectors, is based on the principle of regret minimization, and therefore provides guarantees on the worst-case performance. We demonstrate that our method achieves performance comparable to state-of-the-art methods in several benchmarks while also shielding the user from the burden of blindly selecting the SC and OOD detectors, optimally reducing the worst-case rejection risk.

Cite this Paper

BibTeX
@InProceedings{pmlr-v286-dadalto-camara-gomes25a, title = {Optimal Zero-shot Regret Minimization for Selective Classification with Out-of-Distribution Detection}, author = {Dadalto C\^{a}mara Gomes, Eduardo and Romanelli, Marco}, booktitle = {Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence}, pages = {1497--1520}, year = {2025}, editor = {Chiappa, Silvia and Magliacane, Sara}, volume = {286}, series = {Proceedings of Machine Learning Research}, month = {21--25 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v286/main/assets/dadalto-camara-gomes25a/dadalto-camara-gomes25a.pdf}, url = {https://proceedings.mlr.press/v286/dadalto-camara-gomes25a.html}, abstract = {Selective Classification with Out-of-Distribution Detection (SCOD) is a general framework that combines the detection of incorrectly classified in-distribution samples and out-of-distribution samples. Previous solutions for SCOD heavily rely on the choice of Selective Classification (SC) and Out-of-Distribution (OOD) detectors selected at test time. Notably, the performance of these detectors varies across different underlying data distributions. Hence, a poor choice can affect the efficacy of the SCOD framework. On the other hand, making an informed choice is impossible without samples from both in- and out-distribution. We propose an optimal zero-shot black-box method for SCOD that aggregates off-the-shelf detectors, is based on the principle of regret minimization, and therefore provides guarantees on the worst-case performance. We demonstrate that our method achieves performance comparable to state-of-the-art methods in several benchmarks while also shielding the user from the burden of blindly selecting the SC and OOD detectors, optimally reducing the worst-case rejection risk.} }
Endnote
%0 Conference Paper %T Optimal Zero-shot Regret Minimization for Selective Classification with Out-of-Distribution Detection %A Eduardo Dadalto Câmara Gomes %A Marco Romanelli %B Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2025 %E Silvia Chiappa %E Sara Magliacane %F pmlr-v286-dadalto-camara-gomes25a %I PMLR %P 1497--1520 %U https://proceedings.mlr.press/v286/dadalto-camara-gomes25a.html %V 286 %X Selective Classification with Out-of-Distribution Detection (SCOD) is a general framework that combines the detection of incorrectly classified in-distribution samples and out-of-distribution samples. Previous solutions for SCOD heavily rely on the choice of Selective Classification (SC) and Out-of-Distribution (OOD) detectors selected at test time. Notably, the performance of these detectors varies across different underlying data distributions. Hence, a poor choice can affect the efficacy of the SCOD framework. On the other hand, making an informed choice is impossible without samples from both in- and out-distribution. We propose an optimal zero-shot black-box method for SCOD that aggregates off-the-shelf detectors, is based on the principle of regret minimization, and therefore provides guarantees on the worst-case performance. We demonstrate that our method achieves performance comparable to state-of-the-art methods in several benchmarks while also shielding the user from the burden of blindly selecting the SC and OOD detectors, optimally reducing the worst-case rejection risk.
APA
Dadalto Câmara Gomes, E. & Romanelli, M.. (2025). Optimal Zero-shot Regret Minimization for Selective Classification with Out-of-Distribution Detection. Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 286:1497-1520 Available from https://proceedings.mlr.press/v286/dadalto-camara-gomes25a.html.

Related Material