Faster Recalibration of an Online Predictor via Approachability

Princewill Okoroafor, Bobby Kleinberg, Wen Sun
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:4690-4698, 2024.

Abstract

Predictive models in ML need to be trustworthy and reliable, which often at the very least means outputting calibrated probabilities. This can be particularly difficult to guarantee in the online prediction setting when the outcome sequence can be generated adversarially. In this paper we introduce a technique using Blackwell’s approachability theorem for taking an online predictive model which might not be calibrated and transforming its predictions to calibrated predictions without much increase to the loss of the original model. Our proposed algorithm achieves calibration and accuracy at a faster rate than existing techniques (Kuleshov and Ermon, 2017) and is the first algorithm to offer a flexible tradeoff between calibration error and accuracy in the online setting. We demonstrate this by characterizing the space of jointly achievable calibration and regret using our technique.

Cite this Paper

BibTeX
@InProceedings{pmlr-v238-okoroafor24a, title = {Faster Recalibration of an Online Predictor via Approachability}, author = {Okoroafor, Princewill and Kleinberg, Bobby and Sun, Wen}, booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics}, pages = {4690--4698}, year = {2024}, editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen}, volume = {238}, series = {Proceedings of Machine Learning Research}, month = {02--04 May}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v238/okoroafor24a/okoroafor24a.pdf}, url = {https://proceedings.mlr.press/v238/okoroafor24a.html}, abstract = {Predictive models in ML need to be trustworthy and reliable, which often at the very least means outputting calibrated probabilities. This can be particularly difficult to guarantee in the online prediction setting when the outcome sequence can be generated adversarially. In this paper we introduce a technique using Blackwell’s approachability theorem for taking an online predictive model which might not be calibrated and transforming its predictions to calibrated predictions without much increase to the loss of the original model. Our proposed algorithm achieves calibration and accuracy at a faster rate than existing techniques (Kuleshov and Ermon, 2017) and is the first algorithm to offer a flexible tradeoff between calibration error and accuracy in the online setting. We demonstrate this by characterizing the space of jointly achievable calibration and regret using our technique.} }
Endnote
%0 Conference Paper %T Faster Recalibration of an Online Predictor via Approachability %A Princewill Okoroafor %A Bobby Kleinberg %A Wen Sun %B Proceedings of The 27th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2024 %E Sanjoy Dasgupta %E Stephan Mandt %E Yingzhen Li %F pmlr-v238-okoroafor24a %I PMLR %P 4690--4698 %U https://proceedings.mlr.press/v238/okoroafor24a.html %V 238 %X Predictive models in ML need to be trustworthy and reliable, which often at the very least means outputting calibrated probabilities. This can be particularly difficult to guarantee in the online prediction setting when the outcome sequence can be generated adversarially. In this paper we introduce a technique using Blackwell’s approachability theorem for taking an online predictive model which might not be calibrated and transforming its predictions to calibrated predictions without much increase to the loss of the original model. Our proposed algorithm achieves calibration and accuracy at a faster rate than existing techniques (Kuleshov and Ermon, 2017) and is the first algorithm to offer a flexible tradeoff between calibration error and accuracy in the online setting. We demonstrate this by characterizing the space of jointly achievable calibration and regret using our technique.
APA
Okoroafor, P., Kleinberg, B. & Sun, W.. (2024). Faster Recalibration of an Online Predictor via Approachability. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 238:4690-4698 Available from https://proceedings.mlr.press/v238/okoroafor24a.html.

Related Material