World Guidance: World Modeling in Condition Space for Action Generation

Yue Su, Sijin Chen, Haixin Shi, Mingyu Liu, Zhengshen Zhang, Ningyuan Huang, Weiheng Zhong, Zhengbang Zhu, Yuxiao Liu, Xihui Liu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:115917-115933, 2026.

Abstract

Leveraging future observation modeling to facilitate action generation presents a promising avenue for enhancing the capabilities of Vision-Language-Action (VLA) models. However, existing approaches struggle to strike a balance between maintaining efficient, predictable future representations and preserving sufficient fine-grained information to guide precise action generation. To address this limitation, we propose WoG (World Guidance), a framework that maps future observations into compact conditions by injecting them into the action inference pipeline. The VLA is then trained to simultaneously predict these compressed conditions alongside future actions, thereby achieving effective world modeling within the condition space for action inference. We demonstrate that modeling and predicting this condition space not only facilitates fine-grained action generation but also exhibits superior generalization capabilities. Moreover, it learns effectively from substantial human manipulation videos. Extensive experiments across both simulation and real-world environments validate that WoG significantly outperforms existing methods based on future prediction. Project page is available at: https://selen-suyue.github.io/WoGNet/.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-su26c, title = {World Guidance: World Modeling in Condition Space for Action Generation}, author = {Su, Yue and Chen, Sijin and Shi, Haixin and Liu, Mingyu and Zhang, Zhengshen and Huang, Ningyuan and Zhong, Weiheng and Zhu, Zhengbang and Liu, Yuxiao and Liu, Xihui}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {115917--115933}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/su26c/su26c.pdf}, url = {https://proceedings.mlr.press/v306/su26c.html}, abstract = {Leveraging future observation modeling to facilitate action generation presents a promising avenue for enhancing the capabilities of Vision-Language-Action (VLA) models. However, existing approaches struggle to strike a balance between maintaining efficient, predictable future representations and preserving sufficient fine-grained information to guide precise action generation. To address this limitation, we propose WoG (World Guidance), a framework that maps future observations into compact conditions by injecting them into the action inference pipeline. The VLA is then trained to simultaneously predict these compressed conditions alongside future actions, thereby achieving effective world modeling within the condition space for action inference. We demonstrate that modeling and predicting this condition space not only facilitates fine-grained action generation but also exhibits superior generalization capabilities. Moreover, it learns effectively from substantial human manipulation videos. Extensive experiments across both simulation and real-world environments validate that WoG significantly outperforms existing methods based on future prediction. Project page is available at: https://selen-suyue.github.io/WoGNet/.} }
Endnote
%0 Conference Paper %T World Guidance: World Modeling in Condition Space for Action Generation %A Yue Su %A Sijin Chen %A Haixin Shi %A Mingyu Liu %A Zhengshen Zhang %A Ningyuan Huang %A Weiheng Zhong %A Zhengbang Zhu %A Yuxiao Liu %A Xihui Liu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-su26c %I PMLR %P 115917--115933 %U https://proceedings.mlr.press/v306/su26c.html %V 306 %X Leveraging future observation modeling to facilitate action generation presents a promising avenue for enhancing the capabilities of Vision-Language-Action (VLA) models. However, existing approaches struggle to strike a balance between maintaining efficient, predictable future representations and preserving sufficient fine-grained information to guide precise action generation. To address this limitation, we propose WoG (World Guidance), a framework that maps future observations into compact conditions by injecting them into the action inference pipeline. The VLA is then trained to simultaneously predict these compressed conditions alongside future actions, thereby achieving effective world modeling within the condition space for action inference. We demonstrate that modeling and predicting this condition space not only facilitates fine-grained action generation but also exhibits superior generalization capabilities. Moreover, it learns effectively from substantial human manipulation videos. Extensive experiments across both simulation and real-world environments validate that WoG significantly outperforms existing methods based on future prediction. Project page is available at: https://selen-suyue.github.io/WoGNet/.
APA
Su, Y., Chen, S., Shi, H., Liu, M., Zhang, Z., Huang, N., Zhong, W., Zhu, Z., Liu, Y. & Liu, X.. (2026). World Guidance: World Modeling in Condition Space for Action Generation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:115917-115933 Available from https://proceedings.mlr.press/v306/su26c.html.

Related Material