HiPhO: How Far Are (M)LLMs from Humans in the Latest High School Physics Olympiad Benchmark?

Fangchen Yu, Haiyuan Wan, Qianjia Cheng, Yuchen Zhang, Jiacheng Chen, Fujun Han, Yulun Wu, Junchi Yao, Ruilizhen Hu, Ning Ding, Yu Cheng, Tao Chen, Lei Bai, Dongzhan Zhou, Yun Luo, Ganqu Cui, Peng Ye
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:151221-151250, 2026.

Abstract

Recently, the physics reasoning capabilities of (M)LLMs have attracted growing attention. However, existing physics benchmarks lack systematic coverage of recent physics Olympiads and direct comparison with human contestants. We present HiPhO, the first benchmark dedicated to high school physics Olympiads with human-aligned evaluation. HiPhO highlights three key innovations. (1) Comprehensive data: it compiles 13 latest Olympiads from 2024–2025, covering international and regional competitions and spanning mixed modalities from text-only to diagram-based problems. (2) Professional evaluation: it adopts official rubrics for fine-grained answer- and step-level grading aligned with human examiners. (3) Human-level comparison: it assigns gold, silver, and bronze medals to models based on official medal scores, enabling direct comparison with human contestants. Evaluating 30 (M)LLMs across 13 exams, we find that most open-source MLLMs remain at or below the bronze level, open-source LLMs demonstrate notable progress with multiple gold medals, and closed-source MLLMs achieve 6-13 gold medals, while most models still fall well short of full marks. These results underscore the substantial gap between current (M)LLMs and top human contestants, as well as the room for further improvement. The dataset and leaderboard are available at https://huggingface.co/datasets/SciYu/HiPhO and https://phyarena.github.io, respectively.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-yu26ay, title = {{H}i{P}h{O}: How Far Are ({M}){LLM}s from Humans in the Latest High School Physics Olympiad Benchmark?}, author = {Yu, Fangchen and Wan, Haiyuan and Cheng, Qianjia and Zhang, Yuchen and Chen, Jiacheng and Han, Fujun and Wu, Yulun and Yao, Junchi and Hu, Ruilizhen and Ding, Ning and Cheng, Yu and Chen, Tao and Bai, Lei and Zhou, Dongzhan and Luo, Yun and Cui, Ganqu and Ye, Peng}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {151221--151250}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yu26ay/yu26ay.pdf}, url = {https://proceedings.mlr.press/v306/yu26ay.html}, abstract = {Recently, the physics reasoning capabilities of (M)LLMs have attracted growing attention. However, existing physics benchmarks lack systematic coverage of recent physics Olympiads and direct comparison with human contestants. We present HiPhO, the first benchmark dedicated to high school physics Olympiads with human-aligned evaluation. HiPhO highlights three key innovations. (1) Comprehensive data: it compiles 13 latest Olympiads from 2024–2025, covering international and regional competitions and spanning mixed modalities from text-only to diagram-based problems. (2) Professional evaluation: it adopts official rubrics for fine-grained answer- and step-level grading aligned with human examiners. (3) Human-level comparison: it assigns gold, silver, and bronze medals to models based on official medal scores, enabling direct comparison with human contestants. Evaluating 30 (M)LLMs across 13 exams, we find that most open-source MLLMs remain at or below the bronze level, open-source LLMs demonstrate notable progress with multiple gold medals, and closed-source MLLMs achieve 6-13 gold medals, while most models still fall well short of full marks. These results underscore the substantial gap between current (M)LLMs and top human contestants, as well as the room for further improvement. The dataset and leaderboard are available at https://huggingface.co/datasets/SciYu/HiPhO and https://phyarena.github.io, respectively.} }
Endnote
%0 Conference Paper %T HiPhO: How Far Are (M)LLMs from Humans in the Latest High School Physics Olympiad Benchmark? %A Fangchen Yu %A Haiyuan Wan %A Qianjia Cheng %A Yuchen Zhang %A Jiacheng Chen %A Fujun Han %A Yulun Wu %A Junchi Yao %A Ruilizhen Hu %A Ning Ding %A Yu Cheng %A Tao Chen %A Lei Bai %A Dongzhan Zhou %A Yun Luo %A Ganqu Cui %A Peng Ye %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yu26ay %I PMLR %P 151221--151250 %U https://proceedings.mlr.press/v306/yu26ay.html %V 306 %X Recently, the physics reasoning capabilities of (M)LLMs have attracted growing attention. However, existing physics benchmarks lack systematic coverage of recent physics Olympiads and direct comparison with human contestants. We present HiPhO, the first benchmark dedicated to high school physics Olympiads with human-aligned evaluation. HiPhO highlights three key innovations. (1) Comprehensive data: it compiles 13 latest Olympiads from 2024–2025, covering international and regional competitions and spanning mixed modalities from text-only to diagram-based problems. (2) Professional evaluation: it adopts official rubrics for fine-grained answer- and step-level grading aligned with human examiners. (3) Human-level comparison: it assigns gold, silver, and bronze medals to models based on official medal scores, enabling direct comparison with human contestants. Evaluating 30 (M)LLMs across 13 exams, we find that most open-source MLLMs remain at or below the bronze level, open-source LLMs demonstrate notable progress with multiple gold medals, and closed-source MLLMs achieve 6-13 gold medals, while most models still fall well short of full marks. These results underscore the substantial gap between current (M)LLMs and top human contestants, as well as the room for further improvement. The dataset and leaderboard are available at https://huggingface.co/datasets/SciYu/HiPhO and https://phyarena.github.io, respectively.
APA
Yu, F., Wan, H., Cheng, Q., Zhang, Y., Chen, J., Han, F., Wu, Y., Yao, J., Hu, R., Ding, N., Cheng, Y., Chen, T., Bai, L., Zhou, D., Luo, Y., Cui, G. & Ye, P.. (2026). HiPhO: How Far Are (M)LLMs from Humans in the Latest High School Physics Olympiad Benchmark?. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:151221-151250 Available from https://proceedings.mlr.press/v306/yu26ay.html.

Related Material