Online Calibrated and Conformal Prediction Improves Bayesian Optimization

Shachi Deshpande, Charles Marx, Volodymyr Kuleshov
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:1450-1458, 2024.

Abstract

Accurate uncertainty estimates are important in sequential model-based decision-making tasks such as Bayesian optimization. However, these estimates can be imperfect if the data violates assumptions made by the model (e.g., Gaussianity). This paper studies which uncertainties are needed in model-based decision-making and in Bayesian optimization, and argues that uncertainties can benefit from calibration—i.e., an 80% predictive interval should contain the true outcome 80% of the time. Maintaining calibration, however, can be challenging when the data is non-stationary and depends on our actions. We propose using simple algorithms based on online learning to provably maintain calibration on non-i.i.d. data, and we show how to integrate these algorithms in Bayesian optimization with minimal overhead. Empirically, we find that calibrated Bayesian optimization converges to better optima in fewer steps, and we demonstrate improved performance on standard benchmark functions and hyperparameter optimization tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v238-deshpande24a, title = {Online Calibrated and Conformal Prediction Improves {B}ayesian Optimization}, author = {Deshpande, Shachi and Marx, Charles and Kuleshov, Volodymyr}, booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics}, pages = {1450--1458}, year = {2024}, editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen}, volume = {238}, series = {Proceedings of Machine Learning Research}, month = {02--04 May}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v238/deshpande24a/deshpande24a.pdf}, url = {https://proceedings.mlr.press/v238/deshpande24a.html}, abstract = {Accurate uncertainty estimates are important in sequential model-based decision-making tasks such as Bayesian optimization. However, these estimates can be imperfect if the data violates assumptions made by the model (e.g., Gaussianity). This paper studies which uncertainties are needed in model-based decision-making and in Bayesian optimization, and argues that uncertainties can benefit from calibration—i.e., an 80% predictive interval should contain the true outcome 80% of the time. Maintaining calibration, however, can be challenging when the data is non-stationary and depends on our actions. We propose using simple algorithms based on online learning to provably maintain calibration on non-i.i.d. data, and we show how to integrate these algorithms in Bayesian optimization with minimal overhead. Empirically, we find that calibrated Bayesian optimization converges to better optima in fewer steps, and we demonstrate improved performance on standard benchmark functions and hyperparameter optimization tasks.} }
Endnote
%0 Conference Paper %T Online Calibrated and Conformal Prediction Improves Bayesian Optimization %A Shachi Deshpande %A Charles Marx %A Volodymyr Kuleshov %B Proceedings of The 27th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2024 %E Sanjoy Dasgupta %E Stephan Mandt %E Yingzhen Li %F pmlr-v238-deshpande24a %I PMLR %P 1450--1458 %U https://proceedings.mlr.press/v238/deshpande24a.html %V 238 %X Accurate uncertainty estimates are important in sequential model-based decision-making tasks such as Bayesian optimization. However, these estimates can be imperfect if the data violates assumptions made by the model (e.g., Gaussianity). This paper studies which uncertainties are needed in model-based decision-making and in Bayesian optimization, and argues that uncertainties can benefit from calibration—i.e., an 80% predictive interval should contain the true outcome 80% of the time. Maintaining calibration, however, can be challenging when the data is non-stationary and depends on our actions. We propose using simple algorithms based on online learning to provably maintain calibration on non-i.i.d. data, and we show how to integrate these algorithms in Bayesian optimization with minimal overhead. Empirically, we find that calibrated Bayesian optimization converges to better optima in fewer steps, and we demonstrate improved performance on standard benchmark functions and hyperparameter optimization tasks.
APA
Deshpande, S., Marx, C. & Kuleshov, V.. (2024). Online Calibrated and Conformal Prediction Improves Bayesian Optimization. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 238:1450-1458 Available from https://proceedings.mlr.press/v238/deshpande24a.html.

Related Material