BEDTime: A Unified Benchmark for Automatically Describing Time Series

Medhasweta Sen, Zachary Gottesman, Jiaxing Qiu, C. Bayan Bruss, Nam H Nguyen, Thomas Hartvigsen
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:108947-108982, 2026.

Abstract

Recent works propose complex multi-modal models that handle both time series and language, ultimately claiming high performance on complex tasks like time series reasoning and cross-modal question answering. However, they skip foundational evaluations that such complex models should have mastered. So we ask a simple question:$\textit{How well can recent models describe structural properties of time series?}$ To answer this, we propose that successful models should be able to $\textit{recognize}$, $\textit{differentiate}$, and $\textit{generate}$ descriptions of univariate time series. We then create $\textbf{BEDTime}$, a benchmark to assess these novel tasks, that comprises $\textbf{five datasets}$ reformatted across $\textbf{three modalities}$. In evaluating $\textbf{17 state-of-the-art models}$, we find that (1) surprisingly, dedicated time series-language models fall short, despite being designed for similar tasks, (2) vision language models are quite capable, (3) language only methods perform worst, despite many lauding their potential, and (4) all approaches are clearly fragile to a range of real world robustness tests, indicating directions for future work. Together, our findings critique prior works’ claims and provide avenues for advancing multi-modal time series modeling.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-sen26a, title = {{BEDT}ime: A Unified Benchmark for Automatically Describing Time Series}, author = {Sen, Medhasweta and Gottesman, Zachary and Qiu, Jiaxing and Bruss, C. Bayan and Nguyen, Nam H and Hartvigsen, Thomas}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {108947--108982}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/sen26a/sen26a.pdf}, url = {https://proceedings.mlr.press/v306/sen26a.html}, abstract = {Recent works propose complex multi-modal models that handle both time series and language, ultimately claiming high performance on complex tasks like time series reasoning and cross-modal question answering. However, they skip foundational evaluations that such complex models should have mastered. So we ask a simple question:$\textit{How well can recent models describe structural properties of time series?}$ To answer this, we propose that successful models should be able to $\textit{recognize}$, $\textit{differentiate}$, and $\textit{generate}$ descriptions of univariate time series. We then create $\textbf{BEDTime}$, a benchmark to assess these novel tasks, that comprises $\textbf{five datasets}$ reformatted across $\textbf{three modalities}$. In evaluating $\textbf{17 state-of-the-art models}$, we find that (1) surprisingly, dedicated time series-language models fall short, despite being designed for similar tasks, (2) vision language models are quite capable, (3) language only methods perform worst, despite many lauding their potential, and (4) all approaches are clearly fragile to a range of real world robustness tests, indicating directions for future work. Together, our findings critique prior works’ claims and provide avenues for advancing multi-modal time series modeling.} }
Endnote
%0 Conference Paper %T BEDTime: A Unified Benchmark for Automatically Describing Time Series %A Medhasweta Sen %A Zachary Gottesman %A Jiaxing Qiu %A C. Bayan Bruss %A Nam H Nguyen %A Thomas Hartvigsen %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-sen26a %I PMLR %P 108947--108982 %U https://proceedings.mlr.press/v306/sen26a.html %V 306 %X Recent works propose complex multi-modal models that handle both time series and language, ultimately claiming high performance on complex tasks like time series reasoning and cross-modal question answering. However, they skip foundational evaluations that such complex models should have mastered. So we ask a simple question:$\textit{How well can recent models describe structural properties of time series?}$ To answer this, we propose that successful models should be able to $\textit{recognize}$, $\textit{differentiate}$, and $\textit{generate}$ descriptions of univariate time series. We then create $\textbf{BEDTime}$, a benchmark to assess these novel tasks, that comprises $\textbf{five datasets}$ reformatted across $\textbf{three modalities}$. In evaluating $\textbf{17 state-of-the-art models}$, we find that (1) surprisingly, dedicated time series-language models fall short, despite being designed for similar tasks, (2) vision language models are quite capable, (3) language only methods perform worst, despite many lauding their potential, and (4) all approaches are clearly fragile to a range of real world robustness tests, indicating directions for future work. Together, our findings critique prior works’ claims and provide avenues for advancing multi-modal time series modeling.
APA
Sen, M., Gottesman, Z., Qiu, J., Bruss, C.B., Nguyen, N.H. & Hartvigsen, T.. (2026). BEDTime: A Unified Benchmark for Automatically Describing Time Series. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:108947-108982 Available from https://proceedings.mlr.press/v306/sen26a.html.

Related Material