State space models, emergence, and ergodicity: How many parameters are needed for stable predictions?

Ingvar Ziemann, Nikolai Matni, George Pappas
Proceedings of the 7th Annual Learning for Dynamics \& Control Conference, PMLR 283:1-11, 2025.

Abstract

How many parameters are required for a model to execute a given task? It has been argued that large language models, pre-trained via self-supervised learning, exhibit emergent capabilities such as multi-step reasoning as their number of parameters reach a critical scale. In the present work, we explore whether this phenomenon can analogously be replicated in a simple theoretical model. We show that the problem of learning linear dynamical systems–a simple instance of self-supervised learning–exhibits a corresponding phase transition. Namely, for every non-ergodic linear system there exists a critical threshold such that a learner using fewer parameters than said threshold cannot achieve bounded error for large sequence lengths. Put differently, in our model we find that tasks exhibiting substantial long-range correlation require a certain critical number of parameters–a phenomenon akin to emergence. We also investigate the role of the learner’s parametrization and consider a simple version of a linear dynamical system with hidden state—an imperfectly observed random walk on the real line. For this situation, we show that there exists no learner using a linear filter which can successfully learn the random walk unless the filter length exceeds a certain threshold depending on the effective memory length and horizon of the problem.

Cite this Paper

BibTeX
@InProceedings{pmlr-v283-ziemann25a, title = {State space models, emergence, and ergodicity: How many parameters are needed for stable predictions?}, author = {Ziemann, Ingvar and Matni, Nikolai and Pappas, George}, booktitle = {Proceedings of the 7th Annual Learning for Dynamics \& Control Conference}, pages = {1--11}, year = {2025}, editor = {Ozay, Necmiye and Balzano, Laura and Panagou, Dimitra and Abate, Alessandro}, volume = {283}, series = {Proceedings of Machine Learning Research}, month = {04--06 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v283/main/assets/ziemann25a/ziemann25a.pdf}, url = {https://proceedings.mlr.press/v283/ziemann25a.html}, abstract = {How many parameters are required for a model to execute a given task? It has been argued that large language models, pre-trained via self-supervised learning, exhibit emergent capabilities such as multi-step reasoning as their number of parameters reach a critical scale. In the present work, we explore whether this phenomenon can analogously be replicated in a simple theoretical model. We show that the problem of learning linear dynamical systems–a simple instance of self-supervised learning–exhibits a corresponding phase transition. Namely, for every non-ergodic linear system there exists a critical threshold such that a learner using fewer parameters than said threshold cannot achieve bounded error for large sequence lengths. Put differently, in our model we find that tasks exhibiting substantial long-range correlation require a certain critical number of parameters–a phenomenon akin to emergence. We also investigate the role of the learner’s parametrization and consider a simple version of a linear dynamical system with hidden state—an imperfectly observed random walk on the real line. For this situation, we show that there exists no learner using a linear filter which can successfully learn the random walk unless the filter length exceeds a certain threshold depending on the effective memory length and horizon of the problem.} }
Endnote
%0 Conference Paper %T State space models, emergence, and ergodicity: How many parameters are needed for stable predictions? %A Ingvar Ziemann %A Nikolai Matni %A George Pappas %B Proceedings of the 7th Annual Learning for Dynamics \& Control Conference %C Proceedings of Machine Learning Research %D 2025 %E Necmiye Ozay %E Laura Balzano %E Dimitra Panagou %E Alessandro Abate %F pmlr-v283-ziemann25a %I PMLR %P 1--11 %U https://proceedings.mlr.press/v283/ziemann25a.html %V 283 %X How many parameters are required for a model to execute a given task? It has been argued that large language models, pre-trained via self-supervised learning, exhibit emergent capabilities such as multi-step reasoning as their number of parameters reach a critical scale. In the present work, we explore whether this phenomenon can analogously be replicated in a simple theoretical model. We show that the problem of learning linear dynamical systems–a simple instance of self-supervised learning–exhibits a corresponding phase transition. Namely, for every non-ergodic linear system there exists a critical threshold such that a learner using fewer parameters than said threshold cannot achieve bounded error for large sequence lengths. Put differently, in our model we find that tasks exhibiting substantial long-range correlation require a certain critical number of parameters–a phenomenon akin to emergence. We also investigate the role of the learner’s parametrization and consider a simple version of a linear dynamical system with hidden state—an imperfectly observed random walk on the real line. For this situation, we show that there exists no learner using a linear filter which can successfully learn the random walk unless the filter length exceeds a certain threshold depending on the effective memory length and horizon of the problem.
APA
Ziemann, I., Matni, N. & Pappas, G.. (2025). State space models, emergence, and ergodicity: How many parameters are needed for stable predictions?. Proceedings of the 7th Annual Learning for Dynamics \& Control Conference, in Proceedings of Machine Learning Research 283:1-11 Available from https://proceedings.mlr.press/v283/ziemann25a.html.

Related Material