Right Decisions from Wrong Predictions: A Mechanism Design Alternative to Individual Calibration

Shengjia Zhao, Stefano Ermon
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:2683-2691, 2021.

Abstract

Decision makers often need to rely on imperfect probabilistic forecasts. While average performance metrics are typically available, it is difficult to assess the quality of individual forecasts and the corresponding utilities. To convey confidence about individual predictions to decision-makers, we propose a compensation mechanism ensuring that the forecasted utility matches the actually accrued utility. While a naive scheme to compensate decision-makers for prediction errors can be exploited and might not be sustainable in the long run, we propose a mechanism based on fair bets and online learning that provably cannot be exploited. We demonstrate an application showing how passengers could confidently optimize individual travel plans based on flight delay probabilities estimated by an airline.

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-zhao21a, title = { Right Decisions from Wrong Predictions: A Mechanism Design Alternative to Individual Calibration }, author = {Zhao, Shengjia and Ermon, Stefano}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {2683--2691}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/zhao21a/zhao21a.pdf}, url = {https://proceedings.mlr.press/v130/zhao21a.html}, abstract = { Decision makers often need to rely on imperfect probabilistic forecasts. While average performance metrics are typically available, it is difficult to assess the quality of individual forecasts and the corresponding utilities. To convey confidence about individual predictions to decision-makers, we propose a compensation mechanism ensuring that the forecasted utility matches the actually accrued utility. While a naive scheme to compensate decision-makers for prediction errors can be exploited and might not be sustainable in the long run, we propose a mechanism based on fair bets and online learning that provably cannot be exploited. We demonstrate an application showing how passengers could confidently optimize individual travel plans based on flight delay probabilities estimated by an airline. } }
Endnote
%0 Conference Paper %T Right Decisions from Wrong Predictions: A Mechanism Design Alternative to Individual Calibration %A Shengjia Zhao %A Stefano Ermon %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-zhao21a %I PMLR %P 2683--2691 %U https://proceedings.mlr.press/v130/zhao21a.html %V 130 %X Decision makers often need to rely on imperfect probabilistic forecasts. While average performance metrics are typically available, it is difficult to assess the quality of individual forecasts and the corresponding utilities. To convey confidence about individual predictions to decision-makers, we propose a compensation mechanism ensuring that the forecasted utility matches the actually accrued utility. While a naive scheme to compensate decision-makers for prediction errors can be exploited and might not be sustainable in the long run, we propose a mechanism based on fair bets and online learning that provably cannot be exploited. We demonstrate an application showing how passengers could confidently optimize individual travel plans based on flight delay probabilities estimated by an airline.
APA
Zhao, S. & Ermon, S.. (2021). Right Decisions from Wrong Predictions: A Mechanism Design Alternative to Individual Calibration . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:2683-2691 Available from https://proceedings.mlr.press/v130/zhao21a.html.

Related Material