What You Think is What You See: Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

Haoxi Li, Qinglin Hou, Jianfei Ma, Jinxiang Lai, Tao Han, Sikai Bai, Jingcai Guo, Jie Zhang, Song Guo
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:67405-67430, 2026.

Abstract

To navigate partially observable visual environments, recent VLM agents increasingly internalize world modeling capabilities directly into their policies via explicit CoT reasoning with reinforcement learning (RL). However, mere passive exploitation of reasoning on visited states is insufficient for sparse-reward agentic tasks, as it lacks the epistemic drive to actively uncover the known unknown required for robust generalization. We ask: Can VLM agents actively find signals that challenge and update their internal world model through curiosity-driven exploration? In this work, we propose GLANCE, a unified framework that bridges reasoning and exploration by grounding the agent’s linguistic world model into the stable visual representations of an evolving target network. Crucially, GLANCE leverages the discrepancy between linguistic prediction and visual reality as an intrinsic curiosity signal within reinforcement learning, steering the agent to actively explore areas where its internal model is uncertain. Extensive experiments across a series of agentic tasks show the effectiveness of GLANCE, and demonstrate that aligning what the agent thinks with what the agent sees is key to solving complex or sparse agentic tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-li26bj, title = {What You Think is What You See: Driving Exploration in {VLM} Agents via Visual-Linguistic Curiosity}, author = {Li, Haoxi and Hou, Qinglin and Ma, Jianfei and Lai, Jinxiang and Han, Tao and Bai, Sikai and Guo, Jingcai and Zhang, Jie and Guo, Song}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {67405--67430}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/li26bj/li26bj.pdf}, url = {https://proceedings.mlr.press/v306/li26bj.html}, abstract = {To navigate partially observable visual environments, recent VLM agents increasingly internalize world modeling capabilities directly into their policies via explicit CoT reasoning with reinforcement learning (RL). However, mere passive exploitation of reasoning on visited states is insufficient for sparse-reward agentic tasks, as it lacks the epistemic drive to actively uncover the known unknown required for robust generalization. We ask: Can VLM agents actively find signals that challenge and update their internal world model through curiosity-driven exploration? In this work, we propose GLANCE, a unified framework that bridges reasoning and exploration by grounding the agent’s linguistic world model into the stable visual representations of an evolving target network. Crucially, GLANCE leverages the discrepancy between linguistic prediction and visual reality as an intrinsic curiosity signal within reinforcement learning, steering the agent to actively explore areas where its internal model is uncertain. Extensive experiments across a series of agentic tasks show the effectiveness of GLANCE, and demonstrate that aligning what the agent thinks with what the agent sees is key to solving complex or sparse agentic tasks.} }
Endnote
%0 Conference Paper %T What You Think is What You See: Driving Exploration in VLM Agents via Visual-Linguistic Curiosity %A Haoxi Li %A Qinglin Hou %A Jianfei Ma %A Jinxiang Lai %A Tao Han %A Sikai Bai %A Jingcai Guo %A Jie Zhang %A Song Guo %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-li26bj %I PMLR %P 67405--67430 %U https://proceedings.mlr.press/v306/li26bj.html %V 306 %X To navigate partially observable visual environments, recent VLM agents increasingly internalize world modeling capabilities directly into their policies via explicit CoT reasoning with reinforcement learning (RL). However, mere passive exploitation of reasoning on visited states is insufficient for sparse-reward agentic tasks, as it lacks the epistemic drive to actively uncover the known unknown required for robust generalization. We ask: Can VLM agents actively find signals that challenge and update their internal world model through curiosity-driven exploration? In this work, we propose GLANCE, a unified framework that bridges reasoning and exploration by grounding the agent’s linguistic world model into the stable visual representations of an evolving target network. Crucially, GLANCE leverages the discrepancy between linguistic prediction and visual reality as an intrinsic curiosity signal within reinforcement learning, steering the agent to actively explore areas where its internal model is uncertain. Extensive experiments across a series of agentic tasks show the effectiveness of GLANCE, and demonstrate that aligning what the agent thinks with what the agent sees is key to solving complex or sparse agentic tasks.
APA
Li, H., Hou, Q., Ma, J., Lai, J., Han, T., Bai, S., Guo, J., Zhang, J. & Guo, S.. (2026). What You Think is What You See: Driving Exploration in VLM Agents via Visual-Linguistic Curiosity. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:67405-67430 Available from https://proceedings.mlr.press/v306/li26bj.html.

Related Material