UnHiPPO: Uncertainty-aware Initialization for State Space Models

Marten Lienen, Abdullah Saydemir, Stephan Günnemann
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:37667-37681, 2025.

Abstract

State space models are emerging as a dominant model class for sequence problems with many relying on the HiPPO framework to initialize their dynamics. However, HiPPO fundamentally assumes data to be noise-free; an assumption often violated in practice. We extend the HiPPO theory with measurement noise and derive an uncertainty-aware initialization for state space model dynamics. In our analysis, we interpret HiPPO as a linear stochastic control problem where the data enters as a noise-free control signal. We then reformulate the problem so that the data become noisy outputs of a latent system and arrive at an alternative dynamics initialization that infers the posterior of this latent system from the data without increasing runtime. Our experiments show that our initialization improves the resistance of state-space models to noise both at training and inference time.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-lienen25a, title = {{U}n{H}i{PPO}: Uncertainty-aware Initialization for State Space Models}, author = {Lienen, Marten and Saydemir, Abdullah and G\"{u}nnemann, Stephan}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {37667--37681}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/lienen25a/lienen25a.pdf}, url = {https://proceedings.mlr.press/v267/lienen25a.html}, abstract = {State space models are emerging as a dominant model class for sequence problems with many relying on the HiPPO framework to initialize their dynamics. However, HiPPO fundamentally assumes data to be noise-free; an assumption often violated in practice. We extend the HiPPO theory with measurement noise and derive an uncertainty-aware initialization for state space model dynamics. In our analysis, we interpret HiPPO as a linear stochastic control problem where the data enters as a noise-free control signal. We then reformulate the problem so that the data become noisy outputs of a latent system and arrive at an alternative dynamics initialization that infers the posterior of this latent system from the data without increasing runtime. Our experiments show that our initialization improves the resistance of state-space models to noise both at training and inference time.} }
Endnote
%0 Conference Paper %T UnHiPPO: Uncertainty-aware Initialization for State Space Models %A Marten Lienen %A Abdullah Saydemir %A Stephan Günnemann %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-lienen25a %I PMLR %P 37667--37681 %U https://proceedings.mlr.press/v267/lienen25a.html %V 267 %X State space models are emerging as a dominant model class for sequence problems with many relying on the HiPPO framework to initialize their dynamics. However, HiPPO fundamentally assumes data to be noise-free; an assumption often violated in practice. We extend the HiPPO theory with measurement noise and derive an uncertainty-aware initialization for state space model dynamics. In our analysis, we interpret HiPPO as a linear stochastic control problem where the data enters as a noise-free control signal. We then reformulate the problem so that the data become noisy outputs of a latent system and arrive at an alternative dynamics initialization that infers the posterior of this latent system from the data without increasing runtime. Our experiments show that our initialization improves the resistance of state-space models to noise both at training and inference time.
APA
Lienen, M., Saydemir, A. & Günnemann, S.. (2025). UnHiPPO: Uncertainty-aware Initialization for State Space Models. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:37667-37681 Available from https://proceedings.mlr.press/v267/lienen25a.html.

Related Material