TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding

Lianyu Hu, Ma Xiaoyu, Zeqin Liao, Yang Liu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:45820-45831, 2026.

Abstract

Chain-of-thought (CoT) reasoning has proven effective for enhancing problem-solving in large language models. However, when applied to multimodal LLMs (MLLMs), existing CoT approaches suffer from a fundamental limitation: they perform reasoning entirely in text without accessing visual features during the reasoning process. After initial visual encoding, image information becomes inaccessible, forcing models to reason based solely on whatever was captured in the initial description, which forms a ‘vision-blind reasoning’ paradigm that limits fine-grained visual extraction, error verification, and adaptive attention. We propose Text-Visual Interleaved Chain-of-Thought (TVI-CoT), a framework that enables explicit interleaving of textual reasoning and visual feature access through learnable control tokens ($⟨\text{Think}⟩$, $⟨\text{Look}⟩$, $⟨\text{Answer}⟩$). These tokens allow dynamic switching between reasoning and visual grounding, attending to relevant image regions conditioned on the evolving reasoning state. Experiments on eight benchmarks demonstrate state-of-the-art results among MLLM-based CoT methods and notable performance boost compared to the baseline: +6.1% on MMMU, +3.8% on MathVerse, +3.4% on MathVista, and +3.4% on ScienceQA. Code is available at https://github.com/hulianyuyy/TVI-CoT.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-hu26ba, title = {{TVI}-{C}o{T}: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding}, author = {Hu, Lianyu and Xiaoyu, Ma and Liao, Zeqin and Liu, Yang}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {45820--45831}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/hu26ba/hu26ba.pdf}, url = {https://proceedings.mlr.press/v306/hu26ba.html}, abstract = {Chain-of-thought (CoT) reasoning has proven effective for enhancing problem-solving in large language models. However, when applied to multimodal LLMs (MLLMs), existing CoT approaches suffer from a fundamental limitation: they perform reasoning entirely in text without accessing visual features during the reasoning process. After initial visual encoding, image information becomes inaccessible, forcing models to reason based solely on whatever was captured in the initial description, which forms a ‘vision-blind reasoning’ paradigm that limits fine-grained visual extraction, error verification, and adaptive attention. We propose Text-Visual Interleaved Chain-of-Thought (TVI-CoT), a framework that enables explicit interleaving of textual reasoning and visual feature access through learnable control tokens ($⟨\text{Think}⟩$, $⟨\text{Look}⟩$, $⟨\text{Answer}⟩$). These tokens allow dynamic switching between reasoning and visual grounding, attending to relevant image regions conditioned on the evolving reasoning state. Experiments on eight benchmarks demonstrate state-of-the-art results among MLLM-based CoT methods and notable performance boost compared to the baseline: +6.1% on MMMU, +3.8% on MathVerse, +3.4% on MathVista, and +3.4% on ScienceQA. Code is available at https://github.com/hulianyuyy/TVI-CoT.} }
Endnote
%0 Conference Paper %T TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding %A Lianyu Hu %A Ma Xiaoyu %A Zeqin Liao %A Yang Liu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-hu26ba %I PMLR %P 45820--45831 %U https://proceedings.mlr.press/v306/hu26ba.html %V 306 %X Chain-of-thought (CoT) reasoning has proven effective for enhancing problem-solving in large language models. However, when applied to multimodal LLMs (MLLMs), existing CoT approaches suffer from a fundamental limitation: they perform reasoning entirely in text without accessing visual features during the reasoning process. After initial visual encoding, image information becomes inaccessible, forcing models to reason based solely on whatever was captured in the initial description, which forms a ‘vision-blind reasoning’ paradigm that limits fine-grained visual extraction, error verification, and adaptive attention. We propose Text-Visual Interleaved Chain-of-Thought (TVI-CoT), a framework that enables explicit interleaving of textual reasoning and visual feature access through learnable control tokens ($⟨\text{Think}⟩$, $⟨\text{Look}⟩$, $⟨\text{Answer}⟩$). These tokens allow dynamic switching between reasoning and visual grounding, attending to relevant image regions conditioned on the evolving reasoning state. Experiments on eight benchmarks demonstrate state-of-the-art results among MLLM-based CoT methods and notable performance boost compared to the baseline: +6.1% on MMMU, +3.8% on MathVerse, +3.4% on MathVista, and +3.4% on ScienceQA. Code is available at https://github.com/hulianyuyy/TVI-CoT.
APA
Hu, L., Xiaoyu, M., Liao, Z. & Liu, Y.. (2026). TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:45820-45831 Available from https://proceedings.mlr.press/v306/hu26ba.html.

Related Material