MMU-RAGent: Massive Multi-Modal User-Centric Retrieval Augmented Generation Benchmark

Luo Qi Chan, Tevin Wang, Shuting Wang, Zhihan Zhang, Alfredo Gomez, Prahaladh Chandrahasan, Andy Tang, Lan Yan, Zimeng Qiu, Morteza Ziyadi, Tongshuang Wu, Mona Diab, Akari Asai, Chenyan Xiong, Izat Temiraliev, Serene Cheng, Diji Yang, Yi Zhang, Kun Ran, Marwah Alaofi, Danula Hettiachchi, Chenglong Ma, Khoi Nguyen Dinh Anh, Khoi Vo Nguyen, Sachin Pathiyan Cherumanal, Lida Rashidi, Falk Scholer, Damiano Spina, Shuoqi Sun, Oleg Zendel, Ikuya Yamada, Wataru Ikeda, Ko Yoshida, Mengyu Ye, Hinata Sugimoto, Masatoshi Suzuki, Hisanori Ozaki, Jun Suzuki, Cattalyya Nuengsigkapian, Dmitrii Magas, Quentin Fever, Naziha Aslam
Proceedings of the NeurIPS 2023-2025 Competition Track, PMLR 276:17-52, 2026.

Abstract

We present the results of the MMU-RAGent competition. The competition tasked participants with building systems capable of answering real-world queries drawn from MS MARCO Web Search and Chatbot Arena conversations, as well as queries collected live through the RAG-Arena evaluation platform. To support large-scale retrieval, participants were given API access to the English subsets of ClueWeb22-B and ClueWeb22-A (87M and 800M documents) and AWS-hosted infrastructure for system deployment and evaluation. Submissions were evaluated using a combination of human Likert-scale ratings, live preference judgments collected via RAG-Arena, LLM-as-a-Judge evaluation, and automatic metrics. The competition allowed both open-source and closed-source approaches, including systems leveraging proprietary search APIs or models, with separate leaderboard categories to ensure transparent comparison. Participating systems explored a wide range of RAG architectures, including hierarchical retrieval pipelines, structured document parsing, multi-stage reranking, and iterative generation with citation verification. The competition provides a new benchmark for studying retrieval-augmented generation under realistic deployment conditions and highlights emerging system design patterns for scalable and user-aligned RAG.

Cite this Paper


BibTeX
@InProceedings{pmlr-v276-chan26a, title = {{MMU-RAGent}: Massive Multi-Modal User-Centric Retrieval Augmented Generation Benchmark}, author = {Chan, Luo Qi and Wang, Tevin and Wang, Shuting and Zhang, Zhihan and Gomez, Alfredo and Chandrahasan, Prahaladh and Tang, Andy and Yan, Lan and Qiu, Zimeng and Ziyadi, Morteza and Wu, Tongshuang and Diab, Mona and Asai, Akari and Xiong, Chenyan and Temiraliev, Izat and Cheng, Serene and Yang, Diji and Zhang, Yi and Ran, Kun and Alaofi, Marwah and Hettiachchi, Danula and Ma, Chenglong and Dinh Anh, Khoi Nguyen and Nguyen, Khoi Vo and Pathiyan Cherumanal, Sachin and Rashidi, Lida and Scholer, Falk and Spina, Damiano and Sun, Shuoqi and Zendel, Oleg and Yamada, Ikuya and Ikeda, Wataru and Yoshida, Ko and Ye, Mengyu and Sugimoto, Hinata and Suzuki, Masatoshi and Ozaki, Hisanori and Suzuki, Jun and Nuengsigkapian, Cattalyya and Magas, Dmitrii and Fever, Quentin and Aslam, Naziha}, booktitle = {Proceedings of the NeurIPS 2023-2025 Competition Track}, pages = {17--52}, year = {2026}, editor = {Qin, Tao and Zhang, Kun and Frellsen, Jes}, volume = {276}, series = {Proceedings of Machine Learning Research}, month = {02--07 Dec}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v276/main/assets/chan26a/chan26a.pdf}, url = {https://proceedings.mlr.press/v276/chan26a.html}, abstract = {We present the results of the MMU-RAGent competition. The competition tasked participants with building systems capable of answering real-world queries drawn from MS MARCO Web Search and Chatbot Arena conversations, as well as queries collected live through the RAG-Arena evaluation platform. To support large-scale retrieval, participants were given API access to the English subsets of ClueWeb22-B and ClueWeb22-A (87M and 800M documents) and AWS-hosted infrastructure for system deployment and evaluation. Submissions were evaluated using a combination of human Likert-scale ratings, live preference judgments collected via RAG-Arena, LLM-as-a-Judge evaluation, and automatic metrics. The competition allowed both open-source and closed-source approaches, including systems leveraging proprietary search APIs or models, with separate leaderboard categories to ensure transparent comparison. Participating systems explored a wide range of RAG architectures, including hierarchical retrieval pipelines, structured document parsing, multi-stage reranking, and iterative generation with citation verification. The competition provides a new benchmark for studying retrieval-augmented generation under realistic deployment conditions and highlights emerging system design patterns for scalable and user-aligned RAG.} }
Endnote
%0 Conference Paper %T MMU-RAGent: Massive Multi-Modal User-Centric Retrieval Augmented Generation Benchmark %A Luo Qi Chan %A Tevin Wang %A Shuting Wang %A Zhihan Zhang %A Alfredo Gomez %A Prahaladh Chandrahasan %A Andy Tang %A Lan Yan %A Zimeng Qiu %A Morteza Ziyadi %A Tongshuang Wu %A Mona Diab %A Akari Asai %A Chenyan Xiong %A Izat Temiraliev %A Serene Cheng %A Diji Yang %A Yi Zhang %A Kun Ran %A Marwah Alaofi %A Danula Hettiachchi %A Chenglong Ma %A Khoi Nguyen Dinh Anh %A Khoi Vo Nguyen %A Sachin Pathiyan Cherumanal %A Lida Rashidi %A Falk Scholer %A Damiano Spina %A Shuoqi Sun %A Oleg Zendel %A Ikuya Yamada %A Wataru Ikeda %A Ko Yoshida %A Mengyu Ye %A Hinata Sugimoto %A Masatoshi Suzuki %A Hisanori Ozaki %A Jun Suzuki %A Cattalyya Nuengsigkapian %A Dmitrii Magas %A Quentin Fever %A Naziha Aslam %B Proceedings of the NeurIPS 2023-2025 Competition Track %C Proceedings of Machine Learning Research %D 2026 %E Tao Qin %E Kun Zhang %E Jes Frellsen %F pmlr-v276-chan26a %I PMLR %P 17--52 %U https://proceedings.mlr.press/v276/chan26a.html %V 276 %X We present the results of the MMU-RAGent competition. The competition tasked participants with building systems capable of answering real-world queries drawn from MS MARCO Web Search and Chatbot Arena conversations, as well as queries collected live through the RAG-Arena evaluation platform. To support large-scale retrieval, participants were given API access to the English subsets of ClueWeb22-B and ClueWeb22-A (87M and 800M documents) and AWS-hosted infrastructure for system deployment and evaluation. Submissions were evaluated using a combination of human Likert-scale ratings, live preference judgments collected via RAG-Arena, LLM-as-a-Judge evaluation, and automatic metrics. The competition allowed both open-source and closed-source approaches, including systems leveraging proprietary search APIs or models, with separate leaderboard categories to ensure transparent comparison. Participating systems explored a wide range of RAG architectures, including hierarchical retrieval pipelines, structured document parsing, multi-stage reranking, and iterative generation with citation verification. The competition provides a new benchmark for studying retrieval-augmented generation under realistic deployment conditions and highlights emerging system design patterns for scalable and user-aligned RAG.
APA
Chan, L.Q., Wang, T., Wang, S., Zhang, Z., Gomez, A., Chandrahasan, P., Tang, A., Yan, L., Qiu, Z., Ziyadi, M., Wu, T., Diab, M., Asai, A., Xiong, C., Temiraliev, I., Cheng, S., Yang, D., Zhang, Y., Ran, K., Alaofi, M., Hettiachchi, D., Ma, C., Dinh Anh, K.N., Nguyen, K.V., Pathiyan Cherumanal, S., Rashidi, L., Scholer, F., Spina, D., Sun, S., Zendel, O., Yamada, I., Ikeda, W., Yoshida, K., Ye, M., Sugimoto, H., Suzuki, M., Ozaki, H., Suzuki, J., Nuengsigkapian, C., Magas, D., Fever, Q. & Aslam, N.. (2026). MMU-RAGent: Massive Multi-Modal User-Centric Retrieval Augmented Generation Benchmark. Proceedings of the NeurIPS 2023-2025 Competition Track, in Proceedings of Machine Learning Research 276:17-52 Available from https://proceedings.mlr.press/v276/chan26a.html.

Related Material