Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score

Stefan Haas, Luca Killmaier, Alireza Javanmardi, Eyke Hüllermeier
Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337:1882-1912, 2026.

Abstract

Ordinal classification (OC) arises in high-stakes domains such as medicine and finance, where uncertainty quantification must account for the severity of ordinal errors. Conformal prediction ({CP}) provides distribution-free prediction sets with marginal coverage guarantees; however, its practical effectiveness depends critically on the choice of nonconformity function. We introduce a {CP} method for ordinal classification based on the ranked probability score ({RPS}), a proper scoring rule defined over cumulative predictive distributions. Although it reflects ordinal risk quite naturally, it has largely been neglected in conformal ordinal prediction (COP). When used as a measure of nonconformity, {RPS} yields median-centered contiguous prediction sets by construction. The method is model-agnostic, supports both assessed and grouped ordered categorical outcomes, and permits efficient implementation compared to greedy interval selection procedures. Across multiple ordinal image and tabular datasets, {RPS}-based {CP} produces contiguous prediction sets and strikes a favorable balance between prediction set width and the magnitude of ordinal miscoverage relative to existing {CP} methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v337-haas26a, title = {Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score}, author = {Haas, Stefan and Killmaier, Luca and Javanmardi, Alireza and H\"{u}llermeier, Eyke}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {1882--1912}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/haas26a/haas26a.pdf}, url = {https://proceedings.mlr.press/v337/haas26a.html}, abstract = {Ordinal classification (OC) arises in high-stakes domains such as medicine and finance, where uncertainty quantification must account for the severity of ordinal errors. Conformal prediction ({CP}) provides distribution-free prediction sets with marginal coverage guarantees; however, its practical effectiveness depends critically on the choice of nonconformity function. We introduce a {CP} method for ordinal classification based on the ranked probability score ({RPS}), a proper scoring rule defined over cumulative predictive distributions. Although it reflects ordinal risk quite naturally, it has largely been neglected in conformal ordinal prediction (COP). When used as a measure of nonconformity, {RPS} yields median-centered contiguous prediction sets by construction. The method is model-agnostic, supports both assessed and grouped ordered categorical outcomes, and permits efficient implementation compared to greedy interval selection procedures. Across multiple ordinal image and tabular datasets, {RPS}-based {CP} produces contiguous prediction sets and strikes a favorable balance between prediction set width and the magnitude of ordinal miscoverage relative to existing {CP} methods.} }
Endnote
%0 Conference Paper %T Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score %A Stefan Haas %A Luca Killmaier %A Alireza Javanmardi %A Eyke Hüllermeier %B Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Emilija Perković %E Daniel Malinsky %F pmlr-v337-haas26a %I PMLR %P 1882--1912 %U https://proceedings.mlr.press/v337/haas26a.html %V 337 %X Ordinal classification (OC) arises in high-stakes domains such as medicine and finance, where uncertainty quantification must account for the severity of ordinal errors. Conformal prediction ({CP}) provides distribution-free prediction sets with marginal coverage guarantees; however, its practical effectiveness depends critically on the choice of nonconformity function. We introduce a {CP} method for ordinal classification based on the ranked probability score ({RPS}), a proper scoring rule defined over cumulative predictive distributions. Although it reflects ordinal risk quite naturally, it has largely been neglected in conformal ordinal prediction (COP). When used as a measure of nonconformity, {RPS} yields median-centered contiguous prediction sets by construction. The method is model-agnostic, supports both assessed and grouped ordered categorical outcomes, and permits efficient implementation compared to greedy interval selection procedures. Across multiple ordinal image and tabular datasets, {RPS}-based {CP} produces contiguous prediction sets and strikes a favorable balance between prediction set width and the magnitude of ordinal miscoverage relative to existing {CP} methods.
APA
Haas, S., Killmaier, L., Javanmardi, A. & Hüllermeier, E.. (2026). Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score. Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 337:1882-1912 Available from https://proceedings.mlr.press/v337/haas26a.html.

Related Material