Position: Agent Evaluation Should Be Agentified for Openness, Standardization, and Reproducibility

Xiaoyuan Liu, Tianneng Shi, Wenbo Guo, Dawn Song
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:171365-171374, 2026.

Abstract

Agent systems are advancing quickly, but their evaluation remains fragmented: most benchmarks rely on fixed, LLM-centric harnesses that require heavy integration, create test-production mismatch, and limit fair comparison across diverse agent designs. This position paper argues that the root problem is the lack of an open, agent-agnostic assessment interface. We advocate Agentified Agent Assessment (AAA), where evaluation is also performed by an agent and all participants interact through standardized protocols: A2A for task management and MCP for tool access. Conventional benchmarking defines two separate interfaces, one for the benchmark and one for the agent, while AAA only needs one, separating assessment logic from agent implementation and enabling reproducible, interoperable, and multi-agent evaluation. We further introduce AgentBeats as a concrete realization of AAA: it identifies five practical operation modes that make standardized assessment compatible with real-world constraints on openness, privacy, and reproducibility, and provides recommended practices that allow both agent developers and benchmark designers to adopt AAA with minimal additional effort. Together, AAA and AgentBeats offer a clear path toward open, standardized, and reproducible agent assessment.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-liu26gy, title = {Position: Agent Evaluation Should Be Agentified for Openness, Standardization, and Reproducibility}, author = {Liu, Xiaoyuan and Shi, Tianneng and Guo, Wenbo and Song, Dawn}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {171365--171374}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/liu26gy/liu26gy.pdf}, url = {https://proceedings.mlr.press/v306/liu26gy.html}, abstract = {Agent systems are advancing quickly, but their evaluation remains fragmented: most benchmarks rely on fixed, LLM-centric harnesses that require heavy integration, create test-production mismatch, and limit fair comparison across diverse agent designs. This position paper argues that the root problem is the lack of an open, agent-agnostic assessment interface. We advocate Agentified Agent Assessment (AAA), where evaluation is also performed by an agent and all participants interact through standardized protocols: A2A for task management and MCP for tool access. Conventional benchmarking defines two separate interfaces, one for the benchmark and one for the agent, while AAA only needs one, separating assessment logic from agent implementation and enabling reproducible, interoperable, and multi-agent evaluation. We further introduce AgentBeats as a concrete realization of AAA: it identifies five practical operation modes that make standardized assessment compatible with real-world constraints on openness, privacy, and reproducibility, and provides recommended practices that allow both agent developers and benchmark designers to adopt AAA with minimal additional effort. Together, AAA and AgentBeats offer a clear path toward open, standardized, and reproducible agent assessment.} }
Endnote
%0 Conference Paper %T Position: Agent Evaluation Should Be Agentified for Openness, Standardization, and Reproducibility %A Xiaoyuan Liu %A Tianneng Shi %A Wenbo Guo %A Dawn Song %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-liu26gy %I PMLR %P 171365--171374 %U https://proceedings.mlr.press/v306/liu26gy.html %V 306 %X Agent systems are advancing quickly, but their evaluation remains fragmented: most benchmarks rely on fixed, LLM-centric harnesses that require heavy integration, create test-production mismatch, and limit fair comparison across diverse agent designs. This position paper argues that the root problem is the lack of an open, agent-agnostic assessment interface. We advocate Agentified Agent Assessment (AAA), where evaluation is also performed by an agent and all participants interact through standardized protocols: A2A for task management and MCP for tool access. Conventional benchmarking defines two separate interfaces, one for the benchmark and one for the agent, while AAA only needs one, separating assessment logic from agent implementation and enabling reproducible, interoperable, and multi-agent evaluation. We further introduce AgentBeats as a concrete realization of AAA: it identifies five practical operation modes that make standardized assessment compatible with real-world constraints on openness, privacy, and reproducibility, and provides recommended practices that allow both agent developers and benchmark designers to adopt AAA with minimal additional effort. Together, AAA and AgentBeats offer a clear path toward open, standardized, and reproducible agent assessment.
APA
Liu, X., Shi, T., Guo, W. & Song, D.. (2026). Position: Agent Evaluation Should Be Agentified for Openness, Standardization, and Reproducibility. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:171365-171374 Available from https://proceedings.mlr.press/v306/liu26gy.html.

Related Material