AOEB: Benchmarking Agent-Oriented Multimodal Embeddings

Xin Zhang, Jiaxin Xu, Mengjia Zhou, Xinping Zhao, Yinghui Li, Di Yin, Xing Sun, Meishan Zhang, Baotian Hu, Wenjie Li, Min Zhang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:159666-159688, 2026.

Abstract

LLM agents powered by retrieval and RAG are increasingly prevalent across research and applications. Embedding models play a critical role in these systems, particularly in embedding-based retrieval. However, current benchmarks for embeddings remain focused on general-purpose scenarios, which may fail to align well with the diverse and evolving needs of agentic applications. To close this gap, we introduce Agent-Oriented Embedding Benchmark (AOEB), a comprehensive evaluation suite dedicated to agent-centric retrieval for embedding models. AOEB is characterized by two key features: (1) Multi-Task, covering five essential capabilities for retrieval in LLM agents, including code, tool, reasoning, and memory retrieval; and (2) Multi-Modal, providing evaluation with both textual and visual data for each task category. We evaluate representative embedding models on AOEB and observe that they exhibit distinct strengths across different agent-oriented retrieval tasks. By curating AOEB, we aim to promote a move toward more practically oriented directions within the embedding community and foster further progress.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-zhang26hf, title = {{AOEB}: Benchmarking Agent-Oriented Multimodal Embeddings}, author = {Zhang, Xin and Xu, Jiaxin and Zhou, Mengjia and Zhao, Xinping and Li, Yinghui and Yin, Di and Sun, Xing and Zhang, Meishan and Hu, Baotian and Li, Wenjie and Zhang, Min}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {159666--159688}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhang26hf/zhang26hf.pdf}, url = {https://proceedings.mlr.press/v306/zhang26hf.html}, abstract = {LLM agents powered by retrieval and RAG are increasingly prevalent across research and applications. Embedding models play a critical role in these systems, particularly in embedding-based retrieval. However, current benchmarks for embeddings remain focused on general-purpose scenarios, which may fail to align well with the diverse and evolving needs of agentic applications. To close this gap, we introduce Agent-Oriented Embedding Benchmark (AOEB), a comprehensive evaluation suite dedicated to agent-centric retrieval for embedding models. AOEB is characterized by two key features: (1) Multi-Task, covering five essential capabilities for retrieval in LLM agents, including code, tool, reasoning, and memory retrieval; and (2) Multi-Modal, providing evaluation with both textual and visual data for each task category. We evaluate representative embedding models on AOEB and observe that they exhibit distinct strengths across different agent-oriented retrieval tasks. By curating AOEB, we aim to promote a move toward more practically oriented directions within the embedding community and foster further progress.} }
Endnote
%0 Conference Paper %T AOEB: Benchmarking Agent-Oriented Multimodal Embeddings %A Xin Zhang %A Jiaxin Xu %A Mengjia Zhou %A Xinping Zhao %A Yinghui Li %A Di Yin %A Xing Sun %A Meishan Zhang %A Baotian Hu %A Wenjie Li %A Min Zhang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhang26hf %I PMLR %P 159666--159688 %U https://proceedings.mlr.press/v306/zhang26hf.html %V 306 %X LLM agents powered by retrieval and RAG are increasingly prevalent across research and applications. Embedding models play a critical role in these systems, particularly in embedding-based retrieval. However, current benchmarks for embeddings remain focused on general-purpose scenarios, which may fail to align well with the diverse and evolving needs of agentic applications. To close this gap, we introduce Agent-Oriented Embedding Benchmark (AOEB), a comprehensive evaluation suite dedicated to agent-centric retrieval for embedding models. AOEB is characterized by two key features: (1) Multi-Task, covering five essential capabilities for retrieval in LLM agents, including code, tool, reasoning, and memory retrieval; and (2) Multi-Modal, providing evaluation with both textual and visual data for each task category. We evaluate representative embedding models on AOEB and observe that they exhibit distinct strengths across different agent-oriented retrieval tasks. By curating AOEB, we aim to promote a move toward more practically oriented directions within the embedding community and foster further progress.
APA
Zhang, X., Xu, J., Zhou, M., Zhao, X., Li, Y., Yin, D., Sun, X., Zhang, M., Hu, B., Li, W. & Zhang, M.. (2026). AOEB: Benchmarking Agent-Oriented Multimodal Embeddings. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:159666-159688 Available from https://proceedings.mlr.press/v306/zhang26hf.html.

Related Material