[edit]
Position: Agent Evaluation Should Be Agentified for Openness, Standardization, and Reproducibility
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:171365-171374, 2026.
Abstract
Agent systems are advancing quickly, but their evaluation remains fragmented: most benchmarks rely on fixed, LLM-centric harnesses that require heavy integration, create test-production mismatch, and limit fair comparison across diverse agent designs. This position paper argues that the root problem is the lack of an open, agent-agnostic assessment interface. We advocate Agentified Agent Assessment (AAA), where evaluation is also performed by an agent and all participants interact through standardized protocols: A2A for task management and MCP for tool access. Conventional benchmarking defines two separate interfaces, one for the benchmark and one for the agent, while AAA only needs one, separating assessment logic from agent implementation and enabling reproducible, interoperable, and multi-agent evaluation. We further introduce AgentBeats as a concrete realization of AAA: it identifies five practical operation modes that make standardized assessment compatible with real-world constraints on openness, privacy, and reproducibility, and provides recommended practices that allow both agent developers and benchmark designers to adopt AAA with minimal additional effort. Together, AAA and AgentBeats offer a clear path toward open, standardized, and reproducible agent assessment.