PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception

Yana Wei, Hongbo Peng, Yanlin Lai, Liang Zhao, Kangheng Lin, En Yu, Keyu Lv, Han Zhou, Yin Tang, Haodong Li, Mitt Huang, Hangyu Guo, Jianjian Sun, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Vishal M. Patel
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:133990-134014, 2026.

Abstract

We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the dissonance between benchmark saturation and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 12,000 instance-specific rubrics. These criteria are derived from golden captions that constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) The Reliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2) Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 5% perception deficit between open-source and proprietary frontiers; and (3) Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-wei26u, title = {{P}erception{R}ubrics: Calibrating Multimodal Evaluation to Human Perception}, author = {Wei, Yana and Peng, Hongbo and Lai, Yanlin and Zhao, Liang and Lin, Kangheng and Yu, En and Lv, Keyu and Zhou, Han and Tang, Yin and Li, Haodong and Huang, Mitt and Guo, Hangyu and Sun, Jianjian and Ge, Zheng and Zhang, Xiangyu and Jiang, Daxin and Patel, Vishal M.}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {133990--134014}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wei26u/wei26u.pdf}, url = {https://proceedings.mlr.press/v306/wei26u.html}, abstract = {We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the dissonance between benchmark saturation and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 12,000 instance-specific rubrics. These criteria are derived from golden captions that constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) The Reliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2) Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 5% perception deficit between open-source and proprietary frontiers; and (3) Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.} }
Endnote
%0 Conference Paper %T PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception %A Yana Wei %A Hongbo Peng %A Yanlin Lai %A Liang Zhao %A Kangheng Lin %A En Yu %A Keyu Lv %A Han Zhou %A Yin Tang %A Haodong Li %A Mitt Huang %A Hangyu Guo %A Jianjian Sun %A Zheng Ge %A Xiangyu Zhang %A Daxin Jiang %A Vishal M. Patel %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wei26u %I PMLR %P 133990--134014 %U https://proceedings.mlr.press/v306/wei26u.html %V 306 %X We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the dissonance between benchmark saturation and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 12,000 instance-specific rubrics. These criteria are derived from golden captions that constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) The Reliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2) Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 5% perception deficit between open-source and proprietary frontiers; and (3) Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.
APA
Wei, Y., Peng, H., Lai, Y., Zhao, L., Lin, K., Yu, E., Lv, K., Zhou, H., Tang, Y., Li, H., Huang, M., Guo, H., Sun, J., Ge, Z., Zhang, X., Jiang, D. & Patel, V.M.. (2026). PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:133990-134014 Available from https://proceedings.mlr.press/v306/wei26u.html.

Related Material