Flow Equivariant World Models: Structured Memory for Dynamic Environments

Hansen Lillemark, Benhao Huang, Fangneng Zhan, Yilun Du, T. Anderson Keller
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:73968-74002, 2026.

Abstract

Embodied systems experience the world as ’a symphony of flows’: a combination of many continuous streams of sensory input coupled to self-motion, interwoven with the dynamics of external objects. These sensory streams and the underlying dynamics of the world obey smooth, time-parameterized symmetries which existing world models ignore. Without a memory that respects this structure, partial observability presents a major obstacle to existing methods: each observation reveals only a fraction of the world, while unobserved regions continue to evolve. In this work, we introduce Flow Equivariant World Modeling, a framework that leverages time-parameterized symmetries within a latent memory for stable and accurate dynamics prediction over long horizons. The latent memory shifts and transforms equivariantly with self-motion and inferred external object motion, keeping information about out-of-view regions aligned as time progresses. We demonstrate the advantage of this framework over state-of-the-art diffusion, memory-augmented, and recurrent world model architectures on 2D and 3D partially observed video world modeling benchmarks. More broadly, our results suggest that predictive representations become more powerful when they are organized in line with the temporal and dynamical structure of the world they model. Project page: https://flowequivariantworldmodels.github.io/

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-lillemark26a, title = {Flow Equivariant World Models: Structured Memory for Dynamic Environments}, author = {Lillemark, Hansen and Huang, Benhao and Zhan, Fangneng and Du, Yilun and Keller, T. Anderson}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {73968--74002}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/lillemark26a/lillemark26a.pdf}, url = {https://proceedings.mlr.press/v306/lillemark26a.html}, abstract = {Embodied systems experience the world as ’a symphony of flows’: a combination of many continuous streams of sensory input coupled to self-motion, interwoven with the dynamics of external objects. These sensory streams and the underlying dynamics of the world obey smooth, time-parameterized symmetries which existing world models ignore. Without a memory that respects this structure, partial observability presents a major obstacle to existing methods: each observation reveals only a fraction of the world, while unobserved regions continue to evolve. In this work, we introduce Flow Equivariant World Modeling, a framework that leverages time-parameterized symmetries within a latent memory for stable and accurate dynamics prediction over long horizons. The latent memory shifts and transforms equivariantly with self-motion and inferred external object motion, keeping information about out-of-view regions aligned as time progresses. We demonstrate the advantage of this framework over state-of-the-art diffusion, memory-augmented, and recurrent world model architectures on 2D and 3D partially observed video world modeling benchmarks. More broadly, our results suggest that predictive representations become more powerful when they are organized in line with the temporal and dynamical structure of the world they model. Project page: https://flowequivariantworldmodels.github.io/} }
Endnote
%0 Conference Paper %T Flow Equivariant World Models: Structured Memory for Dynamic Environments %A Hansen Lillemark %A Benhao Huang %A Fangneng Zhan %A Yilun Du %A T. Anderson Keller %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-lillemark26a %I PMLR %P 73968--74002 %U https://proceedings.mlr.press/v306/lillemark26a.html %V 306 %X Embodied systems experience the world as ’a symphony of flows’: a combination of many continuous streams of sensory input coupled to self-motion, interwoven with the dynamics of external objects. These sensory streams and the underlying dynamics of the world obey smooth, time-parameterized symmetries which existing world models ignore. Without a memory that respects this structure, partial observability presents a major obstacle to existing methods: each observation reveals only a fraction of the world, while unobserved regions continue to evolve. In this work, we introduce Flow Equivariant World Modeling, a framework that leverages time-parameterized symmetries within a latent memory for stable and accurate dynamics prediction over long horizons. The latent memory shifts and transforms equivariantly with self-motion and inferred external object motion, keeping information about out-of-view regions aligned as time progresses. We demonstrate the advantage of this framework over state-of-the-art diffusion, memory-augmented, and recurrent world model architectures on 2D and 3D partially observed video world modeling benchmarks. More broadly, our results suggest that predictive representations become more powerful when they are organized in line with the temporal and dynamical structure of the world they model. Project page: https://flowequivariantworldmodels.github.io/
APA
Lillemark, H., Huang, B., Zhan, F., Du, Y. & Keller, T.A.. (2026). Flow Equivariant World Models: Structured Memory for Dynamic Environments. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:73968-74002 Available from https://proceedings.mlr.press/v306/lillemark26a.html.

Related Material