ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models

Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma, Akash Gupta, Samuel Roberts, Ioana Croitoru, Simion-Vlad Bogolin, Jialu Tang, Florian Langer, Vyas Raina, Vatsal Raina, Hanyi Xiong, Vishaal Udandarao, Jingyi Lu, Chen Shiyang, Sam Purkis, Tianshuo Yan, Wenye Lin, Gyungin Shin, Qiaochu Yang, Anh Totti Nguyen, David Atkinson, Alexandru Coca, Mikah Dang, Sebastian Dziadzio, Jakob D. Kunz, Kaiqu Liang, Alexander Lo, Brian Pulfer, Steven Walton, Charig Yang, Kai Han, Samuel Albanie
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:105280-105309, 2026.

Abstract

Large Multimodal Models (LMMs) exhibit shortfalls when interpreting images and, by some measures, have poorer spatial cognition than young children or animals. Despite this, they attain high scores on many popular visual benchmarks, with headroom rapidly eroded by model progress. This creates a need for difficult benchmarks that remain relevant for longer. We introduce ZeroBench—a lightweight visual reasoning benchmark curated using adversarial filtering to be “impossible” for frontier LMMs at its original release, with initial SotA scores of 0% pass@1 and pass@5. We track progress on ZeroBench over the subsequent year, observing SotA reaching 6% pass@5 and 19% pass@5, indicating the potential longevity of the benchmark. We evaluate 46 LMMs on ZeroBench, compare performance to a human baseline, analyse strengths and weaknesses, chart a year of progress in visual capabilities, and publicly release ZeroBench at https://zerobench.github.io/.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-roberts26a, title = {{Z}ero{B}ench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models}, author = {Roberts, Jonathan and Taesiri, Mohammad Reza and Sharma, Ansh and Gupta, Akash and Roberts, Samuel and Croitoru, Ioana and Bogolin, Simion-Vlad and Tang, Jialu and Langer, Florian and Raina, Vyas and Raina, Vatsal and Xiong, Hanyi and Udandarao, Vishaal and Lu, Jingyi and Shiyang, Chen and Purkis, Sam and Yan, Tianshuo and Lin, Wenye and Shin, Gyungin and Yang, Qiaochu and Nguyen, Anh Totti and Atkinson, David and Coca, Alexandru and Dang, Mikah and Dziadzio, Sebastian and Kunz, Jakob D. and Liang, Kaiqu and Lo, Alexander and Pulfer, Brian and Walton, Steven and Yang, Charig and Han, Kai and Albanie, Samuel}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {105280--105309}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/roberts26a/roberts26a.pdf}, url = {https://proceedings.mlr.press/v306/roberts26a.html}, abstract = {Large Multimodal Models (LMMs) exhibit shortfalls when interpreting images and, by some measures, have poorer spatial cognition than young children or animals. Despite this, they attain high scores on many popular visual benchmarks, with headroom rapidly eroded by model progress. This creates a need for difficult benchmarks that remain relevant for longer. We introduce ZeroBench—a lightweight visual reasoning benchmark curated using adversarial filtering to be “impossible” for frontier LMMs at its original release, with initial SotA scores of 0% pass@1 and pass@5. We track progress on ZeroBench over the subsequent year, observing SotA reaching 6% pass@5 and 19% pass@5, indicating the potential longevity of the benchmark. We evaluate 46 LMMs on ZeroBench, compare performance to a human baseline, analyse strengths and weaknesses, chart a year of progress in visual capabilities, and publicly release ZeroBench at https://zerobench.github.io/.} }
Endnote
%0 Conference Paper %T ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models %A Jonathan Roberts %A Mohammad Reza Taesiri %A Ansh Sharma %A Akash Gupta %A Samuel Roberts %A Ioana Croitoru %A Simion-Vlad Bogolin %A Jialu Tang %A Florian Langer %A Vyas Raina %A Vatsal Raina %A Hanyi Xiong %A Vishaal Udandarao %A Jingyi Lu %A Chen Shiyang %A Sam Purkis %A Tianshuo Yan %A Wenye Lin %A Gyungin Shin %A Qiaochu Yang %A Anh Totti Nguyen %A David Atkinson %A Alexandru Coca %A Mikah Dang %A Sebastian Dziadzio %A Jakob D. Kunz %A Kaiqu Liang %A Alexander Lo %A Brian Pulfer %A Steven Walton %A Charig Yang %A Kai Han %A Samuel Albanie %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-roberts26a %I PMLR %P 105280--105309 %U https://proceedings.mlr.press/v306/roberts26a.html %V 306 %X Large Multimodal Models (LMMs) exhibit shortfalls when interpreting images and, by some measures, have poorer spatial cognition than young children or animals. Despite this, they attain high scores on many popular visual benchmarks, with headroom rapidly eroded by model progress. This creates a need for difficult benchmarks that remain relevant for longer. We introduce ZeroBench—a lightweight visual reasoning benchmark curated using adversarial filtering to be “impossible” for frontier LMMs at its original release, with initial SotA scores of 0% pass@1 and pass@5. We track progress on ZeroBench over the subsequent year, observing SotA reaching 6% pass@5 and 19% pass@5, indicating the potential longevity of the benchmark. We evaluate 46 LMMs on ZeroBench, compare performance to a human baseline, analyse strengths and weaknesses, chart a year of progress in visual capabilities, and publicly release ZeroBench at https://zerobench.github.io/.
APA
Roberts, J., Taesiri, M.R., Sharma, A., Gupta, A., Roberts, S., Croitoru, I., Bogolin, S., Tang, J., Langer, F., Raina, V., Raina, V., Xiong, H., Udandarao, V., Lu, J., Shiyang, C., Purkis, S., Yan, T., Lin, W., Shin, G., Yang, Q., Nguyen, A.T., Atkinson, D., Coca, A., Dang, M., Dziadzio, S., Kunz, J.D., Liang, K., Lo, A., Pulfer, B., Walton, S., Yang, C., Han, K. & Albanie, S.. (2026). ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:105280-105309 Available from https://proceedings.mlr.press/v306/roberts26a.html.

Related Material