Fixing the Pitfalls of Probabilistic Time-Series Forecasting Evaluation by Kernel Quadrature

Masaki Adachi, Masahiro Fujisawa, Michael A Osborne
Proceedings of the First International Conference on Probabilistic Numerics, PMLR 271:1-11, 2025.

Abstract

Despite the significance of probabilistic time-series forecasting models, their evaluation metrics often involve intractable integrations. The most widely used metric, the continuous ranked probability score (CRPS), is a strictly proper scoring function; however, its computation requires approximation. We found that popular CRPS estimators—specifically, the quantile-based estimator implemented in the widely used GluonTS library and the probability-weighted moment approximation—both exhibit inherent estimation biases. These biases lead to crude approximations, potentially resulting in improper rankings of forecasting model performance. To address this, we introduced a kernel quadrature approach that leverages an unbiased CRPS estimator and employs cubature construction for scalable computation. Empirically, our approach consistently outperforms the two widely used CRPS estimators.

Cite this Paper

BibTeX
@InProceedings{pmlr-v271-adachi25a, title = {Fixing the Pitfalls of Probabilistic Time-Series Forecasting Evaluation by Kernel Quadrature}, author = {Adachi, Masaki and Fujisawa, Masahiro and Osborne, Michael A}, booktitle = {Proceedings of the First International Conference on Probabilistic Numerics}, pages = {1--11}, year = {2025}, editor = {Kanagawa, Motonobu and Cockayne, Jon and Gessner, Alexandra and Hennig, Philipp}, volume = {271}, series = {Proceedings of Machine Learning Research}, month = {01--03 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v271/main/assets/adachi25a/adachi25a.pdf}, url = {https://proceedings.mlr.press/v271/adachi25a.html}, abstract = {Despite the significance of probabilistic time-series forecasting models, their evaluation metrics often involve intractable integrations. The most widely used metric, the continuous ranked probability score (CRPS), is a strictly proper scoring function; however, its computation requires approximation. We found that popular CRPS estimators—specifically, the quantile-based estimator implemented in the widely used GluonTS library and the probability-weighted moment approximation—both exhibit inherent estimation biases. These biases lead to crude approximations, potentially resulting in improper rankings of forecasting model performance. To address this, we introduced a kernel quadrature approach that leverages an unbiased CRPS estimator and employs cubature construction for scalable computation. Empirically, our approach consistently outperforms the two widely used CRPS estimators.} }
Endnote
%0 Conference Paper %T Fixing the Pitfalls of Probabilistic Time-Series Forecasting Evaluation by Kernel Quadrature %A Masaki Adachi %A Masahiro Fujisawa %A Michael A Osborne %B Proceedings of the First International Conference on Probabilistic Numerics %C Proceedings of Machine Learning Research %D 2025 %E Motonobu Kanagawa %E Jon Cockayne %E Alexandra Gessner %E Philipp Hennig %F pmlr-v271-adachi25a %I PMLR %P 1--11 %U https://proceedings.mlr.press/v271/adachi25a.html %V 271 %X Despite the significance of probabilistic time-series forecasting models, their evaluation metrics often involve intractable integrations. The most widely used metric, the continuous ranked probability score (CRPS), is a strictly proper scoring function; however, its computation requires approximation. We found that popular CRPS estimators—specifically, the quantile-based estimator implemented in the widely used GluonTS library and the probability-weighted moment approximation—both exhibit inherent estimation biases. These biases lead to crude approximations, potentially resulting in improper rankings of forecasting model performance. To address this, we introduced a kernel quadrature approach that leverages an unbiased CRPS estimator and employs cubature construction for scalable computation. Empirically, our approach consistently outperforms the two widely used CRPS estimators.
APA
Adachi, M., Fujisawa, M. & Osborne, M.A.. (2025). Fixing the Pitfalls of Probabilistic Time-Series Forecasting Evaluation by Kernel Quadrature. Proceedings of the First International Conference on Probabilistic Numerics, in Proceedings of Machine Learning Research 271:1-11 Available from https://proceedings.mlr.press/v271/adachi25a.html.

Related Material