MME-Reasoning: A Broad-Spectrum Benchmark for Evaluating Logical Reasoning in MLLMs

Jiakang Yuan, Tianshuo Peng, Yilei Jiang, Yiting Lu, Renrui Zhang, Kaituo Feng, Chaoyou Fu, Tao Chen, Lei Bai, Bo Zhang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:151955-152017, 2026.

Abstract

Logical reasoning is a fundamental aspect of human intelligence and an essential capability for multimodal large language models (MLLMs). Despite the significant advancement in multimodal reasoning, existing benchmarks fail to comprehensively evaluate their reasoning abilities due to the lack of explicit categorization for logical reasoning types and an unclear understanding of reasoning. To address these issues, we introduce MME-Reasoning, a comprehensive benchmark designed to evaluate the reasoning ability of MLLMs, which covers all three types of reasoning (i.e., inductive, deductive, and abductive). We carefully curate the data to ensure that each question effectively evaluates reasoning ability rather than perceptual skills or knowledge breadth, and extend the evaluation protocols to cover the evaluation of diverse questions. Our evaluation reveals substantial limitations of SoTA MLLMs when subjected to holistic assessments of logical reasoning capabilities. Even the most advanced MLLMs show limited performance in comprehensive logical reasoning, with notable performance imbalances across reasoning types. In addition, we conducted an in-depth analysis of approaches such as “thinking mode” and Rule-based RL, which are commonly believed to enhance reasoning abilities. We hope the community can pay more attention to the comprehensive reasoning capabilities of MLLMs instead of only focusing on its subset.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-yuan26f, title = {{MME}-Reasoning: A Broad-Spectrum Benchmark for Evaluating Logical Reasoning in {MLLM}s}, author = {Yuan, Jiakang and Peng, Tianshuo and Jiang, Yilei and Lu, Yiting and Zhang, Renrui and Feng, Kaituo and Fu, Chaoyou and Chen, Tao and Bai, Lei and Zhang, Bo}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {151955--152017}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yuan26f/yuan26f.pdf}, url = {https://proceedings.mlr.press/v306/yuan26f.html}, abstract = {Logical reasoning is a fundamental aspect of human intelligence and an essential capability for multimodal large language models (MLLMs). Despite the significant advancement in multimodal reasoning, existing benchmarks fail to comprehensively evaluate their reasoning abilities due to the lack of explicit categorization for logical reasoning types and an unclear understanding of reasoning. To address these issues, we introduce MME-Reasoning, a comprehensive benchmark designed to evaluate the reasoning ability of MLLMs, which covers all three types of reasoning (i.e., inductive, deductive, and abductive). We carefully curate the data to ensure that each question effectively evaluates reasoning ability rather than perceptual skills or knowledge breadth, and extend the evaluation protocols to cover the evaluation of diverse questions. Our evaluation reveals substantial limitations of SoTA MLLMs when subjected to holistic assessments of logical reasoning capabilities. Even the most advanced MLLMs show limited performance in comprehensive logical reasoning, with notable performance imbalances across reasoning types. In addition, we conducted an in-depth analysis of approaches such as “thinking mode” and Rule-based RL, which are commonly believed to enhance reasoning abilities. We hope the community can pay more attention to the comprehensive reasoning capabilities of MLLMs instead of only focusing on its subset.} }
Endnote
%0 Conference Paper %T MME-Reasoning: A Broad-Spectrum Benchmark for Evaluating Logical Reasoning in MLLMs %A Jiakang Yuan %A Tianshuo Peng %A Yilei Jiang %A Yiting Lu %A Renrui Zhang %A Kaituo Feng %A Chaoyou Fu %A Tao Chen %A Lei Bai %A Bo Zhang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yuan26f %I PMLR %P 151955--152017 %U https://proceedings.mlr.press/v306/yuan26f.html %V 306 %X Logical reasoning is a fundamental aspect of human intelligence and an essential capability for multimodal large language models (MLLMs). Despite the significant advancement in multimodal reasoning, existing benchmarks fail to comprehensively evaluate their reasoning abilities due to the lack of explicit categorization for logical reasoning types and an unclear understanding of reasoning. To address these issues, we introduce MME-Reasoning, a comprehensive benchmark designed to evaluate the reasoning ability of MLLMs, which covers all three types of reasoning (i.e., inductive, deductive, and abductive). We carefully curate the data to ensure that each question effectively evaluates reasoning ability rather than perceptual skills or knowledge breadth, and extend the evaluation protocols to cover the evaluation of diverse questions. Our evaluation reveals substantial limitations of SoTA MLLMs when subjected to holistic assessments of logical reasoning capabilities. Even the most advanced MLLMs show limited performance in comprehensive logical reasoning, with notable performance imbalances across reasoning types. In addition, we conducted an in-depth analysis of approaches such as “thinking mode” and Rule-based RL, which are commonly believed to enhance reasoning abilities. We hope the community can pay more attention to the comprehensive reasoning capabilities of MLLMs instead of only focusing on its subset.
APA
Yuan, J., Peng, T., Jiang, Y., Lu, Y., Zhang, R., Feng, K., Fu, C., Chen, T., Bai, L. & Zhang, B.. (2026). MME-Reasoning: A Broad-Spectrum Benchmark for Evaluating Logical Reasoning in MLLMs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:151955-152017 Available from https://proceedings.mlr.press/v306/yuan26f.html.

Related Material