A Worrying Analysis of Probabilistic Time-series Models for Sales Forecasting

Seungjae Jung, Kyung-Min Kim, Hanock Kwak, Young-Jin Park
Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops, PMLR 137:98-105, 2020.

Abstract

Probabilistic time-series models become popular in the forecasting field as they help to make optimal decisions under uncertainty. Despite the growing interest, a lack of thorough analysis hinders choosing what is worth applying for the desired task. In this paper, we analyze the performance of three prominent probabilistic time-series models for sales forecasting. To remove the role of random chance in architecture’s performance, we make two experimental principles; 1) Large-scale dataset with various cross-validation sets. 2) A standardized training and hyperparameter selection. The experimental results show that a simple Multi- layer Perceptron and Linear Regression outperform the probabilistic models on RMSE without any feature engineering. Overall, the probabilistic models fail to achieve better performance on point estimation, such as RMSE and MAPE, than comparably simple baselines. We analyze and discuss the performances of probabilistic time-series models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v137-jung20a, title = {A Worrying Analysis of Probabilistic Time-series Models for Sales Forecasting}, author = {Jung, Seungjae and Kim, Kyung-Min and Kwak, Hanock and Park, Young-Jin}, booktitle = {Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops}, pages = {98--105}, year = {2020}, editor = {Zosa Forde, Jessica and Ruiz, Francisco and Pradier, Melanie F. and Schein, Aaron}, volume = {137}, series = {Proceedings of Machine Learning Research}, month = {12 Dec}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v137/jung20a/jung20a.pdf}, url = {https://proceedings.mlr.press/v137/jung20a.html}, abstract = {Probabilistic time-series models become popular in the forecasting field as they help to make optimal decisions under uncertainty. Despite the growing interest, a lack of thorough analysis hinders choosing what is worth applying for the desired task. In this paper, we analyze the performance of three prominent probabilistic time-series models for sales forecasting. To remove the role of random chance in architecture’s performance, we make two experimental principles; 1) Large-scale dataset with various cross-validation sets. 2) A standardized training and hyperparameter selection. The experimental results show that a simple Multi- layer Perceptron and Linear Regression outperform the probabilistic models on RMSE without any feature engineering. Overall, the probabilistic models fail to achieve better performance on point estimation, such as RMSE and MAPE, than comparably simple baselines. We analyze and discuss the performances of probabilistic time-series models.} }
Endnote
%0 Conference Paper %T A Worrying Analysis of Probabilistic Time-series Models for Sales Forecasting %A Seungjae Jung %A Kyung-Min Kim %A Hanock Kwak %A Young-Jin Park %B Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops %C Proceedings of Machine Learning Research %D 2020 %E Jessica Zosa Forde %E Francisco Ruiz %E Melanie F. Pradier %E Aaron Schein %F pmlr-v137-jung20a %I PMLR %P 98--105 %U https://proceedings.mlr.press/v137/jung20a.html %V 137 %X Probabilistic time-series models become popular in the forecasting field as they help to make optimal decisions under uncertainty. Despite the growing interest, a lack of thorough analysis hinders choosing what is worth applying for the desired task. In this paper, we analyze the performance of three prominent probabilistic time-series models for sales forecasting. To remove the role of random chance in architecture’s performance, we make two experimental principles; 1) Large-scale dataset with various cross-validation sets. 2) A standardized training and hyperparameter selection. The experimental results show that a simple Multi- layer Perceptron and Linear Regression outperform the probabilistic models on RMSE without any feature engineering. Overall, the probabilistic models fail to achieve better performance on point estimation, such as RMSE and MAPE, than comparably simple baselines. We analyze and discuss the performances of probabilistic time-series models.
APA
Jung, S., Kim, K., Kwak, H. & Park, Y.. (2020). A Worrying Analysis of Probabilistic Time-series Models for Sales Forecasting. Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops, in Proceedings of Machine Learning Research 137:98-105 Available from https://proceedings.mlr.press/v137/jung20a.html.

Related Material