Factored Latent Action World Models

Zizhao Wang, Chang Shi, Jiaheng Hu, Kevin Rohling, Roberto Martı́n-Martı́n, Amy Zhang, Peter Stone
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:129753-129775, 2026.

Abstract

Learning latent actions from action-free video has emerged as a powerful paradigm for scaling up controllable world model learning. Latent actions provide a natural interface for users to iteratively generate and manipulate videos. However, most existing approaches rely on monolithic inverse and forward dynamics models that learn a single latent action to control the entire scene, and therefore struggle in complex environments where multiple entities act simultaneously. This paper introduces Factored Latent Action Model (FLAM), a factored dynamics framework that decomposes the scene into independent factors, each inferring its own latent action and predicting its own next-step factor value. This factorized structure enables more accurate modeling of complex multi-entity dynamics and improves video generation quality in action-free video settings compared to monolithic models. Based on experiments on both simulation and real-world multi-entity datasets, we find that FLAM outperforms prior work in prediction accuracy and representation quality, and facilitates downstream policy learning, demonstrating the benefits of factorized latent action models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-wang26gs, title = {Factored Latent Action World Models}, author = {Wang, Zizhao and Shi, Chang and Hu, Jiaheng and Rohling, Kevin and Mart\'{\i}n-Mart\'{\i}n, Roberto and Zhang, Amy and Stone, Peter}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {129753--129775}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wang26gs/wang26gs.pdf}, url = {https://proceedings.mlr.press/v306/wang26gs.html}, abstract = {Learning latent actions from action-free video has emerged as a powerful paradigm for scaling up controllable world model learning. Latent actions provide a natural interface for users to iteratively generate and manipulate videos. However, most existing approaches rely on monolithic inverse and forward dynamics models that learn a single latent action to control the entire scene, and therefore struggle in complex environments where multiple entities act simultaneously. This paper introduces Factored Latent Action Model (FLAM), a factored dynamics framework that decomposes the scene into independent factors, each inferring its own latent action and predicting its own next-step factor value. This factorized structure enables more accurate modeling of complex multi-entity dynamics and improves video generation quality in action-free video settings compared to monolithic models. Based on experiments on both simulation and real-world multi-entity datasets, we find that FLAM outperforms prior work in prediction accuracy and representation quality, and facilitates downstream policy learning, demonstrating the benefits of factorized latent action models.} }
Endnote
%0 Conference Paper %T Factored Latent Action World Models %A Zizhao Wang %A Chang Shi %A Jiaheng Hu %A Kevin Rohling %A Roberto Martı́n-Martı́n %A Amy Zhang %A Peter Stone %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wang26gs %I PMLR %P 129753--129775 %U https://proceedings.mlr.press/v306/wang26gs.html %V 306 %X Learning latent actions from action-free video has emerged as a powerful paradigm for scaling up controllable world model learning. Latent actions provide a natural interface for users to iteratively generate and manipulate videos. However, most existing approaches rely on monolithic inverse and forward dynamics models that learn a single latent action to control the entire scene, and therefore struggle in complex environments where multiple entities act simultaneously. This paper introduces Factored Latent Action Model (FLAM), a factored dynamics framework that decomposes the scene into independent factors, each inferring its own latent action and predicting its own next-step factor value. This factorized structure enables more accurate modeling of complex multi-entity dynamics and improves video generation quality in action-free video settings compared to monolithic models. Based on experiments on both simulation and real-world multi-entity datasets, we find that FLAM outperforms prior work in prediction accuracy and representation quality, and facilitates downstream policy learning, demonstrating the benefits of factorized latent action models.
APA
Wang, Z., Shi, C., Hu, J., Rohling, K., Martı́n-Martı́n, R., Zhang, A. & Stone, P.. (2026). Factored Latent Action World Models. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:129753-129775 Available from https://proceedings.mlr.press/v306/wang26gs.html.

Related Material