Position: Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility

Jialun Cao, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li, Mingwei Liu, Ruixi Qiao, Yuting Han, Chaozheng Wang, Boxi Yu, Pinjia He, Shuai Wang, Zibin Zheng, Michael R. Lyu, Shing-Chi Cheung
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:169590-169655, 2026.

Abstract

Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In the past few years, awareness of benchmark quality has grown. Yet, after a decade-scale (2014 - 2025) survey over 672 code benchmarks, we observed a lag between growing awareness and actual practice. For example, in 2025 alone, the number of benchmarks that ignore code coverage when providing test cases nearly matches the total count accumulated across the previous ten years. In response, we take a clear position: Code benchmarks must prioritize rigor in benchmark construction, reliability in evaluation, and reproducibility in release. To operationalize this position, we introduce a code benchmark guideline HOW2BENCH with 55 checklists. Finally, our further human study also exposed that the current issues not only stem from the significant effort required, but also from a lack of awareness regarding their importance.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-cao26ag, title = {Position: Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility}, author = {Cao, Jialun and Chan, Yuk-Kit and Ling, Zixuan and Wang, Wenxuan and Li, Shuqing and Liu, Mingwei and Qiao, Ruixi and Han, Yuting and Wang, Chaozheng and Yu, Boxi and He, Pinjia and Wang, Shuai and Zheng, Zibin and Lyu, Michael R. and Cheung, Shing-Chi}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {169590--169655}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/cao26ag/cao26ag.pdf}, url = {https://proceedings.mlr.press/v306/cao26ag.html}, abstract = {Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In the past few years, awareness of benchmark quality has grown. Yet, after a decade-scale (2014 - 2025) survey over 672 code benchmarks, we observed a lag between growing awareness and actual practice. For example, in 2025 alone, the number of benchmarks that ignore code coverage when providing test cases nearly matches the total count accumulated across the previous ten years. In response, we take a clear position: Code benchmarks must prioritize rigor in benchmark construction, reliability in evaluation, and reproducibility in release. To operationalize this position, we introduce a code benchmark guideline HOW2BENCH with 55 checklists. Finally, our further human study also exposed that the current issues not only stem from the significant effort required, but also from a lack of awareness regarding their importance.} }
Endnote
%0 Conference Paper %T Position: Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility %A Jialun Cao %A Yuk-Kit Chan %A Zixuan Ling %A Wenxuan Wang %A Shuqing Li %A Mingwei Liu %A Ruixi Qiao %A Yuting Han %A Chaozheng Wang %A Boxi Yu %A Pinjia He %A Shuai Wang %A Zibin Zheng %A Michael R. Lyu %A Shing-Chi Cheung %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-cao26ag %I PMLR %P 169590--169655 %U https://proceedings.mlr.press/v306/cao26ag.html %V 306 %X Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In the past few years, awareness of benchmark quality has grown. Yet, after a decade-scale (2014 - 2025) survey over 672 code benchmarks, we observed a lag between growing awareness and actual practice. For example, in 2025 alone, the number of benchmarks that ignore code coverage when providing test cases nearly matches the total count accumulated across the previous ten years. In response, we take a clear position: Code benchmarks must prioritize rigor in benchmark construction, reliability in evaluation, and reproducibility in release. To operationalize this position, we introduce a code benchmark guideline HOW2BENCH with 55 checklists. Finally, our further human study also exposed that the current issues not only stem from the significant effort required, but also from a lack of awareness regarding their importance.
APA
Cao, J., Chan, Y., Ling, Z., Wang, W., Li, S., Liu, M., Qiao, R., Han, Y., Wang, C., Yu, B., He, P., Wang, S., Zheng, Z., Lyu, M.R. & Cheung, S.. (2026). Position: Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:169590-169655 Available from https://proceedings.mlr.press/v306/cao26ag.html.

Related Material