Conformal Predictive Decision Making: A Comparative Study with Bayesian and Point-Predictive Methods

Simon Lanngren, Martin Toremark, Johan Hallberg Szabadváry
Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 266:379-404, 2025.

Abstract

In many real-world settings, machine learning predictions serve as intermediate outputs used to inform decision-making. However, quantifying and accounting for uncertainty in these decisions remains a fundamental challenge. Conformal Predictive Decision Making is a framework for decision-making under uncertainty that leverages Conformal Predictive Distributions to optimize outcomes over a specified utility function. In this work, we evaluate Conformal Predictive Decision Making on synthetic datasets in both online and inductive settings, and compare its performance to two alternative approaches: Bayesian Decision Theory and Point Predictive Decision Making. Online Conformal Predictive Decision Making showed signs of greater robustness than Bayesian Decision Theory and Point Predictive Decision Making in scenarios involving noisy data and skewed utility functions, suggesting it may be a suitable option in more complex settings. However, it generally performed worse than the two alternative methods. In contrast, inductive Conformal Predictive Decision Making consistently outperformed the alternatives. This, combined with its computational advantages, makes it a promising approach for larger real-world decision-making applications where well-calibrated uncertainty quantification is needed for robustness.

Cite this Paper

BibTeX
@InProceedings{pmlr-v266-lanngren25a, title = {Conformal Predictive Decision Making: A Comparative Study with Bayesian and Point-Predictive Methods}, author = {Lanngren, Simon and Toremark, Martin and Hallberg Szabadv\'{a}ry, Johan}, booktitle = {Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {379--404}, year = {2025}, editor = {Nguyen, Khuong An and Luo, Zhiyuan and Papadopoulos, Harris and Löfström, Tuwe and Carlsson, Lars and Boström, Henrik}, volume = {266}, series = {Proceedings of Machine Learning Research}, month = {10--12 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v266/main/assets/lanngren25a/lanngren25a.pdf}, url = {https://proceedings.mlr.press/v266/lanngren25a.html}, abstract = {In many real-world settings, machine learning predictions serve as intermediate outputs used to inform decision-making. However, quantifying and accounting for uncertainty in these decisions remains a fundamental challenge. Conformal Predictive Decision Making is a framework for decision-making under uncertainty that leverages Conformal Predictive Distributions to optimize outcomes over a specified utility function. In this work, we evaluate Conformal Predictive Decision Making on synthetic datasets in both online and inductive settings, and compare its performance to two alternative approaches: Bayesian Decision Theory and Point Predictive Decision Making. Online Conformal Predictive Decision Making showed signs of greater robustness than Bayesian Decision Theory and Point Predictive Decision Making in scenarios involving noisy data and skewed utility functions, suggesting it may be a suitable option in more complex settings. However, it generally performed worse than the two alternative methods. In contrast, inductive Conformal Predictive Decision Making consistently outperformed the alternatives. This, combined with its computational advantages, makes it a promising approach for larger real-world decision-making applications where well-calibrated uncertainty quantification is needed for robustness.} }
Endnote
%0 Conference Paper %T Conformal Predictive Decision Making: A Comparative Study with Bayesian and Point-Predictive Methods %A Simon Lanngren %A Martin Toremark %A Johan Hallberg Szabadváry %B Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2025 %E Khuong An Nguyen %E Zhiyuan Luo %E Harris Papadopoulos %E Tuwe Löfström %E Lars Carlsson %E Henrik Boström %F pmlr-v266-lanngren25a %I PMLR %P 379--404 %U https://proceedings.mlr.press/v266/lanngren25a.html %V 266 %X In many real-world settings, machine learning predictions serve as intermediate outputs used to inform decision-making. However, quantifying and accounting for uncertainty in these decisions remains a fundamental challenge. Conformal Predictive Decision Making is a framework for decision-making under uncertainty that leverages Conformal Predictive Distributions to optimize outcomes over a specified utility function. In this work, we evaluate Conformal Predictive Decision Making on synthetic datasets in both online and inductive settings, and compare its performance to two alternative approaches: Bayesian Decision Theory and Point Predictive Decision Making. Online Conformal Predictive Decision Making showed signs of greater robustness than Bayesian Decision Theory and Point Predictive Decision Making in scenarios involving noisy data and skewed utility functions, suggesting it may be a suitable option in more complex settings. However, it generally performed worse than the two alternative methods. In contrast, inductive Conformal Predictive Decision Making consistently outperformed the alternatives. This, combined with its computational advantages, makes it a promising approach for larger real-world decision-making applications where well-calibrated uncertainty quantification is needed for robustness.
APA
Lanngren, S., Toremark, M. & Hallberg Szabadváry, J.. (2025). Conformal Predictive Decision Making: A Comparative Study with Bayesian and Point-Predictive Methods. Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 266:379-404 Available from https://proceedings.mlr.press/v266/lanngren25a.html.

Related Material