Future Dynamic 3D Reconstruction: Toward 3D World Modeling with Disentangled Ego-Motion

Nils Morbitzer, Jonathan Evers, Artem Savkin, Thomas Stauner, Nassir Navab, Federico Tombari, Stefano Gasperini
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:90232-90250, 2026.

Abstract

Forecasting the evolution of dynamic environments is crucial for autonomous agents. While generative world models have achieved high photorealism in 2D video synthesis by mixing ego-motion and environmental dynamics within the image plane, they exhibit physical inconsistencies, such as morphing or vanishing objects, especially over long time horizons. In this paper, we propose FR3D, a world-modeling approach that predicts a persistent 3D latent representation for future dynamic 3D reconstruction. Unlike prior works that treat the world as a sequence of image-based features, FR3D explicitly decouples the 3D evolution of the scene from the agent’s trajectory, treating the inferred ego-motion as a latent proxy for action. This disentanglement resolves ambiguities between self-motion and world-motion, ensuring geometric consistency into the future. Furthermore, we introduce a teacher-student distillation strategy that leverages the spatial "common sense" of off-the-shelf foundation models, leading to robust zero-shot generalization. Extensive experiments demonstrate FR3D’s strong performance for future dynamic 3D reconstruction from monocular observations across multiple datasets, even 2 seconds into the future. Project page: https://fr3d-wm.github.io.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-morbitzer26a, title = {Future Dynamic 3{D} Reconstruction: Toward 3{D} World Modeling with Disentangled Ego-Motion}, author = {Morbitzer, Nils and Evers, Jonathan and Savkin, Artem and Stauner, Thomas and Navab, Nassir and Tombari, Federico and Gasperini, Stefano}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {90232--90250}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/morbitzer26a/morbitzer26a.pdf}, url = {https://proceedings.mlr.press/v306/morbitzer26a.html}, abstract = {Forecasting the evolution of dynamic environments is crucial for autonomous agents. While generative world models have achieved high photorealism in 2D video synthesis by mixing ego-motion and environmental dynamics within the image plane, they exhibit physical inconsistencies, such as morphing or vanishing objects, especially over long time horizons. In this paper, we propose FR3D, a world-modeling approach that predicts a persistent 3D latent representation for future dynamic 3D reconstruction. Unlike prior works that treat the world as a sequence of image-based features, FR3D explicitly decouples the 3D evolution of the scene from the agent’s trajectory, treating the inferred ego-motion as a latent proxy for action. This disentanglement resolves ambiguities between self-motion and world-motion, ensuring geometric consistency into the future. Furthermore, we introduce a teacher-student distillation strategy that leverages the spatial "common sense" of off-the-shelf foundation models, leading to robust zero-shot generalization. Extensive experiments demonstrate FR3D’s strong performance for future dynamic 3D reconstruction from monocular observations across multiple datasets, even 2 seconds into the future. Project page: https://fr3d-wm.github.io.} }
Endnote
%0 Conference Paper %T Future Dynamic 3D Reconstruction: Toward 3D World Modeling with Disentangled Ego-Motion %A Nils Morbitzer %A Jonathan Evers %A Artem Savkin %A Thomas Stauner %A Nassir Navab %A Federico Tombari %A Stefano Gasperini %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-morbitzer26a %I PMLR %P 90232--90250 %U https://proceedings.mlr.press/v306/morbitzer26a.html %V 306 %X Forecasting the evolution of dynamic environments is crucial for autonomous agents. While generative world models have achieved high photorealism in 2D video synthesis by mixing ego-motion and environmental dynamics within the image plane, they exhibit physical inconsistencies, such as morphing or vanishing objects, especially over long time horizons. In this paper, we propose FR3D, a world-modeling approach that predicts a persistent 3D latent representation for future dynamic 3D reconstruction. Unlike prior works that treat the world as a sequence of image-based features, FR3D explicitly decouples the 3D evolution of the scene from the agent’s trajectory, treating the inferred ego-motion as a latent proxy for action. This disentanglement resolves ambiguities between self-motion and world-motion, ensuring geometric consistency into the future. Furthermore, we introduce a teacher-student distillation strategy that leverages the spatial "common sense" of off-the-shelf foundation models, leading to robust zero-shot generalization. Extensive experiments demonstrate FR3D’s strong performance for future dynamic 3D reconstruction from monocular observations across multiple datasets, even 2 seconds into the future. Project page: https://fr3d-wm.github.io.
APA
Morbitzer, N., Evers, J., Savkin, A., Stauner, T., Navab, N., Tombari, F. & Gasperini, S.. (2026). Future Dynamic 3D Reconstruction: Toward 3D World Modeling with Disentangled Ego-Motion. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:90232-90250 Available from https://proceedings.mlr.press/v306/morbitzer26a.html.

Related Material