Evaluation of updating strategies for conformal predictive systems in the presence of extreme events

Hugo Werner, Lars Carlsson, Ernst Ahlberg, Henrik Boström
Proceedings of the Tenth Symposium on Conformal and Probabilistic Prediction and Applications, PMLR 152:229-242, 2021.

Abstract

Six different strategies for updating split conformal predictive systems in an online (streaming) setting are evaluated. The updating strategies vary in the extent and frequency of retraining as well as in how training data is split into proper training and calibration sets. An empirical evaluation is presented, considering passenger booking data from a ferry company, which stretches over a number of years. The passenger volumes have changed drastically during 2020 due to COVID-19 and part of the evaluation is focusing on which updating strategies work best under such circumstances. Some strategies are observed to outperform others with respect to continuous ranked probability score and validity, highlighting the potential value of choosing a proper strategy.

Cite this Paper

BibTeX
@InProceedings{pmlr-v152-werner21a, title = {Evaluation of updating strategies for conformal predictive systems in the presence of extreme events}, author = {Werner, Hugo and Carlsson, Lars and Ahlberg, Ernst and Bostr\"{o}m, Henrik}, booktitle = {Proceedings of the Tenth Symposium on Conformal and Probabilistic Prediction and Applications}, pages = {229--242}, year = {2021}, editor = {Carlsson, Lars and Luo, Zhiyuan and Cherubin, Giovanni and An Nguyen, Khuong}, volume = {152}, series = {Proceedings of Machine Learning Research}, month = {08--10 Sep}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v152/werner21a/werner21a.pdf}, url = {https://proceedings.mlr.press/v152/werner21a.html}, abstract = {Six different strategies for updating split conformal predictive systems in an online (streaming) setting are evaluated. The updating strategies vary in the extent and frequency of retraining as well as in how training data is split into proper training and calibration sets. An empirical evaluation is presented, considering passenger booking data from a ferry company, which stretches over a number of years. The passenger volumes have changed drastically during 2020 due to COVID-19 and part of the evaluation is focusing on which updating strategies work best under such circumstances. Some strategies are observed to outperform others with respect to continuous ranked probability score and validity, highlighting the potential value of choosing a proper strategy.} }
Endnote
%0 Conference Paper %T Evaluation of updating strategies for conformal predictive systems in the presence of extreme events %A Hugo Werner %A Lars Carlsson %A Ernst Ahlberg %A Henrik Boström %B Proceedings of the Tenth Symposium on Conformal and Probabilistic Prediction and Applications %C Proceedings of Machine Learning Research %D 2021 %E Lars Carlsson %E Zhiyuan Luo %E Giovanni Cherubin %E Khuong An Nguyen %F pmlr-v152-werner21a %I PMLR %P 229--242 %U https://proceedings.mlr.press/v152/werner21a.html %V 152 %X Six different strategies for updating split conformal predictive systems in an online (streaming) setting are evaluated. The updating strategies vary in the extent and frequency of retraining as well as in how training data is split into proper training and calibration sets. An empirical evaluation is presented, considering passenger booking data from a ferry company, which stretches over a number of years. The passenger volumes have changed drastically during 2020 due to COVID-19 and part of the evaluation is focusing on which updating strategies work best under such circumstances. Some strategies are observed to outperform others with respect to continuous ranked probability score and validity, highlighting the potential value of choosing a proper strategy.
APA
Werner, H., Carlsson, L., Ahlberg, E. & Boström, H.. (2021). Evaluation of updating strategies for conformal predictive systems in the presence of extreme events. Proceedings of the Tenth Symposium on Conformal and Probabilistic Prediction and Applications, in Proceedings of Machine Learning Research 152:229-242 Available from https://proceedings.mlr.press/v152/werner21a.html.

Related Material