Rethinking Visual Intelligence: Insights from Video Pretraining

Pablo Acuaviva, Aram Davtyan, Mariam Hassan, Sebastian Stapf, Ahmad Rahimi, Alexandre Alahi, Paolo Favaro
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:435-472, 2026.

Abstract

Large language models (LLMs) have demonstrated that large-scale pretraining enables systems to adapt rapidly to new problems with little supervision in the language domain. This success, however, has not translated as effectively to the visual domain, where models, including LLMs, continue to struggle with compositional understanding, sample efficiency, and general-purpose problem-solving. We investigate Video Diffusion Models (VDMs) as a promising direction for bridging this gap. Pretraining on spatiotemporal data endows these models with strong inductive biases for structure and dynamics, which we hypothesize can support broad task adaptability. To test this, we design a controlled evaluation in which both a pretrained LLM and a pretrained VDM are equipped with lightweight adapters and presented with tasks in their natural modalities. Across benchmarks including ARC-AGI, ConceptARC, visual games, route planning, and cellular automata, VDMs demonstrate higher data efficiency than their language counterparts. Taken together, our results indicate that video pretraining offers inductive biases that support progress toward visual foundation models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-acuaviva26a, title = {Rethinking Visual Intelligence: Insights from Video Pretraining}, author = {Acuaviva, Pablo and Davtyan, Aram and Hassan, Mariam and Stapf, Sebastian and Rahimi, Ahmad and Alahi, Alexandre and Favaro, Paolo}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {435--472}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/acuaviva26a/acuaviva26a.pdf}, url = {https://proceedings.mlr.press/v306/acuaviva26a.html}, abstract = {Large language models (LLMs) have demonstrated that large-scale pretraining enables systems to adapt rapidly to new problems with little supervision in the language domain. This success, however, has not translated as effectively to the visual domain, where models, including LLMs, continue to struggle with compositional understanding, sample efficiency, and general-purpose problem-solving. We investigate Video Diffusion Models (VDMs) as a promising direction for bridging this gap. Pretraining on spatiotemporal data endows these models with strong inductive biases for structure and dynamics, which we hypothesize can support broad task adaptability. To test this, we design a controlled evaluation in which both a pretrained LLM and a pretrained VDM are equipped with lightweight adapters and presented with tasks in their natural modalities. Across benchmarks including ARC-AGI, ConceptARC, visual games, route planning, and cellular automata, VDMs demonstrate higher data efficiency than their language counterparts. Taken together, our results indicate that video pretraining offers inductive biases that support progress toward visual foundation models.} }
Endnote
%0 Conference Paper %T Rethinking Visual Intelligence: Insights from Video Pretraining %A Pablo Acuaviva %A Aram Davtyan %A Mariam Hassan %A Sebastian Stapf %A Ahmad Rahimi %A Alexandre Alahi %A Paolo Favaro %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-acuaviva26a %I PMLR %P 435--472 %U https://proceedings.mlr.press/v306/acuaviva26a.html %V 306 %X Large language models (LLMs) have demonstrated that large-scale pretraining enables systems to adapt rapidly to new problems with little supervision in the language domain. This success, however, has not translated as effectively to the visual domain, where models, including LLMs, continue to struggle with compositional understanding, sample efficiency, and general-purpose problem-solving. We investigate Video Diffusion Models (VDMs) as a promising direction for bridging this gap. Pretraining on spatiotemporal data endows these models with strong inductive biases for structure and dynamics, which we hypothesize can support broad task adaptability. To test this, we design a controlled evaluation in which both a pretrained LLM and a pretrained VDM are equipped with lightweight adapters and presented with tasks in their natural modalities. Across benchmarks including ARC-AGI, ConceptARC, visual games, route planning, and cellular automata, VDMs demonstrate higher data efficiency than their language counterparts. Taken together, our results indicate that video pretraining offers inductive biases that support progress toward visual foundation models.
APA
Acuaviva, P., Davtyan, A., Hassan, M., Stapf, S., Rahimi, A., Alahi, A. & Favaro, P.. (2026). Rethinking Visual Intelligence: Insights from Video Pretraining. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:435-472 Available from https://proceedings.mlr.press/v306/acuaviva26a.html.

Related Material