Seeing to Generalize: How Visual Data Corrects Binding Shortcuts

Nicolás Buzeta, Felipe Del Rio, Cristian Hinostroza, Denis Parra, Hans Lobel, Rodrigo Toro Icarte
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:10341-10371, 2026.

Abstract

Vision Language Models (VLMs) are designed to extend Large Language Models (LLMs) with visual capabilities, yet in this work we observe a surprising phenomenon: VLMs can outperform their underlying LLMs on purely text-only tasks, particularly in long-context information retrieval. To investigate this effect, we build a controlled synthetic retrieval task and find that a transformer trained only on text achieves perfect in-distribution accuracy but fails to generalize out-of-distribution (OOD), while subsequent training on an image-tokenized version of the same task nearly doubles text-only OOD performance. Mechanistic interpretability reveals that visual training changes the model’s internal binding strategy: text-only training encourages positional shortcuts, whereas image-based training disrupts them through spatial translation invariance and other encoder-level inductive biases, forcing the model to adopt a more robust symbolic binding mechanism that persists even after text-only examples are reintroduced. We further characterize how binding strategies vary across training regimes and visual encoders, and show that analogous shifts occur during pre-trained LLM-to-VLM transitions. Our findings suggest that cross-modal training can enhance reasoning and generalization even for tasks grounded in a single modality.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-buzeta26a, title = {Seeing to Generalize: How Visual Data Corrects Binding Shortcuts}, author = {Buzeta, Nicol\'{a}s and Del Rio, Felipe and Hinostroza, Cristian and Parra, Denis and Lobel, Hans and Icarte, Rodrigo Toro}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {10341--10371}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/buzeta26a/buzeta26a.pdf}, url = {https://proceedings.mlr.press/v306/buzeta26a.html}, abstract = {Vision Language Models (VLMs) are designed to extend Large Language Models (LLMs) with visual capabilities, yet in this work we observe a surprising phenomenon: VLMs can outperform their underlying LLMs on purely text-only tasks, particularly in long-context information retrieval. To investigate this effect, we build a controlled synthetic retrieval task and find that a transformer trained only on text achieves perfect in-distribution accuracy but fails to generalize out-of-distribution (OOD), while subsequent training on an image-tokenized version of the same task nearly doubles text-only OOD performance. Mechanistic interpretability reveals that visual training changes the model’s internal binding strategy: text-only training encourages positional shortcuts, whereas image-based training disrupts them through spatial translation invariance and other encoder-level inductive biases, forcing the model to adopt a more robust symbolic binding mechanism that persists even after text-only examples are reintroduced. We further characterize how binding strategies vary across training regimes and visual encoders, and show that analogous shifts occur during pre-trained LLM-to-VLM transitions. Our findings suggest that cross-modal training can enhance reasoning and generalization even for tasks grounded in a single modality.} }
Endnote
%0 Conference Paper %T Seeing to Generalize: How Visual Data Corrects Binding Shortcuts %A Nicolás Buzeta %A Felipe Del Rio %A Cristian Hinostroza %A Denis Parra %A Hans Lobel %A Rodrigo Toro Icarte %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-buzeta26a %I PMLR %P 10341--10371 %U https://proceedings.mlr.press/v306/buzeta26a.html %V 306 %X Vision Language Models (VLMs) are designed to extend Large Language Models (LLMs) with visual capabilities, yet in this work we observe a surprising phenomenon: VLMs can outperform their underlying LLMs on purely text-only tasks, particularly in long-context information retrieval. To investigate this effect, we build a controlled synthetic retrieval task and find that a transformer trained only on text achieves perfect in-distribution accuracy but fails to generalize out-of-distribution (OOD), while subsequent training on an image-tokenized version of the same task nearly doubles text-only OOD performance. Mechanistic interpretability reveals that visual training changes the model’s internal binding strategy: text-only training encourages positional shortcuts, whereas image-based training disrupts them through spatial translation invariance and other encoder-level inductive biases, forcing the model to adopt a more robust symbolic binding mechanism that persists even after text-only examples are reintroduced. We further characterize how binding strategies vary across training regimes and visual encoders, and show that analogous shifts occur during pre-trained LLM-to-VLM transitions. Our findings suggest that cross-modal training can enhance reasoning and generalization even for tasks grounded in a single modality.
APA
Buzeta, N., Del Rio, F., Hinostroza, C., Parra, D., Lobel, H. & Icarte, R.T.. (2026). Seeing to Generalize: How Visual Data Corrects Binding Shortcuts. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:10341-10371 Available from https://proceedings.mlr.press/v306/buzeta26a.html.

Related Material