AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation

Yunxiao Shi, Wujiang Xu, Tingwei Chen, Haoning Shang, Ling Yang, Yunfeng Wan, Zhuo Cao, Xing Zi, Dimitris N. Metaxas, Min Xu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:112338-112358, 2026.

Abstract

LLM agents are rapidly becoming a practical interface for task automation, yet selecting suitable deployable configurations remains underexplored. Existing LLM leaderboards and tool/agent benchmarks evaluate components in isolation and are fragmented across tasks, metrics, and candidate pools, leaving a critical research gap: there is little query-conditioned supervision for learning to recommend end-to-end compositional agent configurations. We introduce AgentSelect, a benchmark that reframes agent selection as narrative query-to-agent recommendation over capability profiles. AgentSelect systematically converts heterogeneous evaluation artifacts into unified positive-only interaction data, comprising 111,179 queries, 107,721 deployable agents, and 251,103 interaction records from 40+ sources across LLM-only, toolkit-only, and compositional agents. Our analyses reveal a shift from dense head reuse to long-tail, near one-off supervision, where ID-based interaction methods become fragile and reliable recommendation increasingly requires content-aware intent-to-capability matching. We validate the synthesized supervision through counterfactual, ablation, and show practical transferability on the MuleRun agent marketplace and a small-scale end to end Agno deployment study. Overall, AgentSelect provides the first unified data and evaluation infrastructure for agent recommendation, which establishes a reproducible foundation to study and accelerate the emerging agent ecosystem. The resources are available at [https://github.com/Ancientshi/AgentSelect].

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-shi26ac, title = {{A}gent{S}elect: Benchmark for Narrative Query-to-Agent Recommendation}, author = {Shi, Yunxiao and Xu, Wujiang and Chen, Tingwei and Shang, Haoning and Yang, Ling and Wan, Yunfeng and Cao, Zhuo and Zi, Xing and Metaxas, Dimitris N. and Xu, Min}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {112338--112358}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/shi26ac/shi26ac.pdf}, url = {https://proceedings.mlr.press/v306/shi26ac.html}, abstract = {LLM agents are rapidly becoming a practical interface for task automation, yet selecting suitable deployable configurations remains underexplored. Existing LLM leaderboards and tool/agent benchmarks evaluate components in isolation and are fragmented across tasks, metrics, and candidate pools, leaving a critical research gap: there is little query-conditioned supervision for learning to recommend end-to-end compositional agent configurations. We introduce AgentSelect, a benchmark that reframes agent selection as narrative query-to-agent recommendation over capability profiles. AgentSelect systematically converts heterogeneous evaluation artifacts into unified positive-only interaction data, comprising 111,179 queries, 107,721 deployable agents, and 251,103 interaction records from 40+ sources across LLM-only, toolkit-only, and compositional agents. Our analyses reveal a shift from dense head reuse to long-tail, near one-off supervision, where ID-based interaction methods become fragile and reliable recommendation increasingly requires content-aware intent-to-capability matching. We validate the synthesized supervision through counterfactual, ablation, and show practical transferability on the MuleRun agent marketplace and a small-scale end to end Agno deployment study. Overall, AgentSelect provides the first unified data and evaluation infrastructure for agent recommendation, which establishes a reproducible foundation to study and accelerate the emerging agent ecosystem. The resources are available at [https://github.com/Ancientshi/AgentSelect].} }
Endnote
%0 Conference Paper %T AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation %A Yunxiao Shi %A Wujiang Xu %A Tingwei Chen %A Haoning Shang %A Ling Yang %A Yunfeng Wan %A Zhuo Cao %A Xing Zi %A Dimitris N. Metaxas %A Min Xu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-shi26ac %I PMLR %P 112338--112358 %U https://proceedings.mlr.press/v306/shi26ac.html %V 306 %X LLM agents are rapidly becoming a practical interface for task automation, yet selecting suitable deployable configurations remains underexplored. Existing LLM leaderboards and tool/agent benchmarks evaluate components in isolation and are fragmented across tasks, metrics, and candidate pools, leaving a critical research gap: there is little query-conditioned supervision for learning to recommend end-to-end compositional agent configurations. We introduce AgentSelect, a benchmark that reframes agent selection as narrative query-to-agent recommendation over capability profiles. AgentSelect systematically converts heterogeneous evaluation artifacts into unified positive-only interaction data, comprising 111,179 queries, 107,721 deployable agents, and 251,103 interaction records from 40+ sources across LLM-only, toolkit-only, and compositional agents. Our analyses reveal a shift from dense head reuse to long-tail, near one-off supervision, where ID-based interaction methods become fragile and reliable recommendation increasingly requires content-aware intent-to-capability matching. We validate the synthesized supervision through counterfactual, ablation, and show practical transferability on the MuleRun agent marketplace and a small-scale end to end Agno deployment study. Overall, AgentSelect provides the first unified data and evaluation infrastructure for agent recommendation, which establishes a reproducible foundation to study and accelerate the emerging agent ecosystem. The resources are available at [https://github.com/Ancientshi/AgentSelect].
APA
Shi, Y., Xu, W., Chen, T., Shang, H., Yang, L., Wan, Y., Cao, Z., Zi, X., Metaxas, D.N. & Xu, M.. (2026). AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:112338-112358 Available from https://proceedings.mlr.press/v306/shi26ac.html.

Related Material