CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding

Hanjun Luo, Chiming Ni, Jiaheng Wen, Zhimu Huang, Bingduo Liao, Sylvia Chung, Yiran Wang, Yingbin Jin, Jialin Li, Xinfeng Li, Wenyuan Xu, Xiaofeng Wang, Hanan Salam
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:82662-82703, 2026.

Abstract

LLM-powered coding agents are reshaping the development paradigm. However, existing evaluation systems, neither traditional tests for humans nor benchmarks for LLMs, fail to capture this shift, excluding problems that require both human reasoning to guide solutions and AI efficiency for implementation. We introduce CentaurEval, a unified, ecologically valid benchmark for measuring human-in-the-loop value in coding. CentaurEval’s core innovation is its "Collaboration-Necessary" problem templates, which are intractable for standalone LLMs or humans, but solvable through effective collaboration. CentaurEval dynamically instantiates tasks from 45 templates, providing a standardized IDE for humans and a reproducible 450-task toolkit for LLMs. We benchmark 45 participants against 5 LLMs under 4 levels of human intervention. Results show that while LLMs or humans alone achieve poor pass rates (0.67% and 18.89%), human–AI collaboration significantly improves to 31.11%. Our analysis reveals an emerging co-reasoning partnership, challenging the traditional human-tool hierarchy by showing that strategic breakthroughs can originate from either humans or AI.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-luo26t, title = {{C}entaur{E}val: Benchmarking Human-in-the-Loop Value in Agentic Coding}, author = {Luo, Hanjun and Ni, Chiming and Wen, Jiaheng and Huang, Zhimu and Liao, Bingduo and Chung, Sylvia and Wang, Yiran and Jin, Yingbin and Li, Jialin and Li, Xinfeng and Xu, Wenyuan and Wang, Xiaofeng and Salam, Hanan}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {82662--82703}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/luo26t/luo26t.pdf}, url = {https://proceedings.mlr.press/v306/luo26t.html}, abstract = {LLM-powered coding agents are reshaping the development paradigm. However, existing evaluation systems, neither traditional tests for humans nor benchmarks for LLMs, fail to capture this shift, excluding problems that require both human reasoning to guide solutions and AI efficiency for implementation. We introduce CentaurEval, a unified, ecologically valid benchmark for measuring human-in-the-loop value in coding. CentaurEval’s core innovation is its "Collaboration-Necessary" problem templates, which are intractable for standalone LLMs or humans, but solvable through effective collaboration. CentaurEval dynamically instantiates tasks from 45 templates, providing a standardized IDE for humans and a reproducible 450-task toolkit for LLMs. We benchmark 45 participants against 5 LLMs under 4 levels of human intervention. Results show that while LLMs or humans alone achieve poor pass rates (0.67% and 18.89%), human–AI collaboration significantly improves to 31.11%. Our analysis reveals an emerging co-reasoning partnership, challenging the traditional human-tool hierarchy by showing that strategic breakthroughs can originate from either humans or AI.} }
Endnote
%0 Conference Paper %T CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding %A Hanjun Luo %A Chiming Ni %A Jiaheng Wen %A Zhimu Huang %A Bingduo Liao %A Sylvia Chung %A Yiran Wang %A Yingbin Jin %A Jialin Li %A Xinfeng Li %A Wenyuan Xu %A Xiaofeng Wang %A Hanan Salam %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-luo26t %I PMLR %P 82662--82703 %U https://proceedings.mlr.press/v306/luo26t.html %V 306 %X LLM-powered coding agents are reshaping the development paradigm. However, existing evaluation systems, neither traditional tests for humans nor benchmarks for LLMs, fail to capture this shift, excluding problems that require both human reasoning to guide solutions and AI efficiency for implementation. We introduce CentaurEval, a unified, ecologically valid benchmark for measuring human-in-the-loop value in coding. CentaurEval’s core innovation is its "Collaboration-Necessary" problem templates, which are intractable for standalone LLMs or humans, but solvable through effective collaboration. CentaurEval dynamically instantiates tasks from 45 templates, providing a standardized IDE for humans and a reproducible 450-task toolkit for LLMs. We benchmark 45 participants against 5 LLMs under 4 levels of human intervention. Results show that while LLMs or humans alone achieve poor pass rates (0.67% and 18.89%), human–AI collaboration significantly improves to 31.11%. Our analysis reveals an emerging co-reasoning partnership, challenging the traditional human-tool hierarchy by showing that strategic breakthroughs can originate from either humans or AI.
APA
Luo, H., Ni, C., Wen, J., Huang, Z., Liao, B., Chung, S., Wang, Y., Jin, Y., Li, J., Li, X., Xu, W., Wang, X. & Salam, H.. (2026). CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:82662-82703 Available from https://proceedings.mlr.press/v306/luo26t.html.

Related Material