Position: Your VLM May Not Be Thinking with Interleaved Images

Wenjie Yang, Siqi Zhu, Zengfeng Huang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:172794-172806, 2026.

Abstract

"Thinking with images" has emerged as a central research theme in the realm of Vision-Language Models (VLMs). This multimodal reasoning paradigm typically features interleaved images generated via tool use or code execution as part of the Chain-of-Thought (CoT). While reinforcement learning (RL) has driven impressive performance within this paradigm, in this position paper, we argue that current VLMs seldom truly "think" with interleaved images. Through empirical evidence and analysis, we demonstrate that interleaved images do not play a significant role in the success of recent "Thinking with images" methods. Instead, the primary source of performance gains is the improved language generation distribution resulting from fine-tuning. These findings challenge the prevailing belief that "Thinking with images" VLMs actively utilize visual information to complete visual tasks. To improve mechanistic transparency, we suggest that future "Thinking with images" works include lightweight ablation studies to verify the necessity of interleaved images. Furthermore, we call upon the community to develop fundamentally novel benchmarks and advocate for more informative visual tools.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-yang26ed, title = {Position: Your {VLM} May Not Be Thinking with Interleaved Images}, author = {Yang, Wenjie and Zhu, Siqi and Huang, Zengfeng}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {172794--172806}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yang26ed/yang26ed.pdf}, url = {https://proceedings.mlr.press/v306/yang26ed.html}, abstract = {"Thinking with images" has emerged as a central research theme in the realm of Vision-Language Models (VLMs). This multimodal reasoning paradigm typically features interleaved images generated via tool use or code execution as part of the Chain-of-Thought (CoT). While reinforcement learning (RL) has driven impressive performance within this paradigm, in this position paper, we argue that current VLMs seldom truly "think" with interleaved images. Through empirical evidence and analysis, we demonstrate that interleaved images do not play a significant role in the success of recent "Thinking with images" methods. Instead, the primary source of performance gains is the improved language generation distribution resulting from fine-tuning. These findings challenge the prevailing belief that "Thinking with images" VLMs actively utilize visual information to complete visual tasks. To improve mechanistic transparency, we suggest that future "Thinking with images" works include lightweight ablation studies to verify the necessity of interleaved images. Furthermore, we call upon the community to develop fundamentally novel benchmarks and advocate for more informative visual tools.} }
Endnote
%0 Conference Paper %T Position: Your VLM May Not Be Thinking with Interleaved Images %A Wenjie Yang %A Siqi Zhu %A Zengfeng Huang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yang26ed %I PMLR %P 172794--172806 %U https://proceedings.mlr.press/v306/yang26ed.html %V 306 %X "Thinking with images" has emerged as a central research theme in the realm of Vision-Language Models (VLMs). This multimodal reasoning paradigm typically features interleaved images generated via tool use or code execution as part of the Chain-of-Thought (CoT). While reinforcement learning (RL) has driven impressive performance within this paradigm, in this position paper, we argue that current VLMs seldom truly "think" with interleaved images. Through empirical evidence and analysis, we demonstrate that interleaved images do not play a significant role in the success of recent "Thinking with images" methods. Instead, the primary source of performance gains is the improved language generation distribution resulting from fine-tuning. These findings challenge the prevailing belief that "Thinking with images" VLMs actively utilize visual information to complete visual tasks. To improve mechanistic transparency, we suggest that future "Thinking with images" works include lightweight ablation studies to verify the necessity of interleaved images. Furthermore, we call upon the community to develop fundamentally novel benchmarks and advocate for more informative visual tools.
APA
Yang, W., Zhu, S. & Huang, Z.. (2026). Position: Your VLM May Not Be Thinking with Interleaved Images. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:172794-172806 Available from https://proceedings.mlr.press/v306/yang26ed.html.

Related Material