Position: Current Benchmarking Hinders Real Progress in Deep Learning for Time Series Forecasting

Valentina Moretti, Ivan Marisca, Cesare Alippi, Andrea Cini
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:171515-171539, 2026.

Abstract

Deep learning models have grown popular in time series applications. However, the large quantity of newly proposed architectures and the often contradictory empirical results make it difficult to assess which design choice and model component drives performance. In this position paper, we argue that current benchmarking practices fail to identify the factors responsible for performance differences, thus slowing down progress in the field. In particular, differences in crucial design dimensions are overlooked when comparing architectures, ultimately leading to inconsistent outcomes. To support our position, we show that such differences—often treated as mere implementation details—can have a greater impact than adopting specific sequence modeling layers. We discuss how overlooked aspects(such as globality and locality) can (1) fundamentally change the class of the forecasting method and (2) drastically affect empirical results. Our findings suggest rethinking our benchmarking practices and focusing on the foundational aspects of the forecasting problem when designing and comparing architectures. As a concrete step, we propose an auxiliary forecasting model card, i.e., a template with a set of fields to characterize existing and new forecasting architectures based on key design choices.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-moretti26a, title = {Position: Current Benchmarking Hinders Real Progress in Deep Learning for Time Series Forecasting}, author = {Moretti, Valentina and Marisca, Ivan and Alippi, Cesare and Cini, Andrea}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {171515--171539}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/moretti26a/moretti26a.pdf}, url = {https://proceedings.mlr.press/v306/moretti26a.html}, abstract = {Deep learning models have grown popular in time series applications. However, the large quantity of newly proposed architectures and the often contradictory empirical results make it difficult to assess which design choice and model component drives performance. In this position paper, we argue that current benchmarking practices fail to identify the factors responsible for performance differences, thus slowing down progress in the field. In particular, differences in crucial design dimensions are overlooked when comparing architectures, ultimately leading to inconsistent outcomes. To support our position, we show that such differences—often treated as mere implementation details—can have a greater impact than adopting specific sequence modeling layers. We discuss how overlooked aspects(such as globality and locality) can (1) fundamentally change the class of the forecasting method and (2) drastically affect empirical results. Our findings suggest rethinking our benchmarking practices and focusing on the foundational aspects of the forecasting problem when designing and comparing architectures. As a concrete step, we propose an auxiliary forecasting model card, i.e., a template with a set of fields to characterize existing and new forecasting architectures based on key design choices.} }
Endnote
%0 Conference Paper %T Position: Current Benchmarking Hinders Real Progress in Deep Learning for Time Series Forecasting %A Valentina Moretti %A Ivan Marisca %A Cesare Alippi %A Andrea Cini %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-moretti26a %I PMLR %P 171515--171539 %U https://proceedings.mlr.press/v306/moretti26a.html %V 306 %X Deep learning models have grown popular in time series applications. However, the large quantity of newly proposed architectures and the often contradictory empirical results make it difficult to assess which design choice and model component drives performance. In this position paper, we argue that current benchmarking practices fail to identify the factors responsible for performance differences, thus slowing down progress in the field. In particular, differences in crucial design dimensions are overlooked when comparing architectures, ultimately leading to inconsistent outcomes. To support our position, we show that such differences—often treated as mere implementation details—can have a greater impact than adopting specific sequence modeling layers. We discuss how overlooked aspects(such as globality and locality) can (1) fundamentally change the class of the forecasting method and (2) drastically affect empirical results. Our findings suggest rethinking our benchmarking practices and focusing on the foundational aspects of the forecasting problem when designing and comparing architectures. As a concrete step, we propose an auxiliary forecasting model card, i.e., a template with a set of fields to characterize existing and new forecasting architectures based on key design choices.
APA
Moretti, V., Marisca, I., Alippi, C. & Cini, A.. (2026). Position: Current Benchmarking Hinders Real Progress in Deep Learning for Time Series Forecasting. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:171515-171539 Available from https://proceedings.mlr.press/v306/moretti26a.html.

Related Material