[edit]
MMU-RAGent: Massive Multi-Modal User-Centric Retrieval Augmented Generation Benchmark
Proceedings of the NeurIPS 2023-2025 Competition Track, PMLR 276:17-52, 2026.
Abstract
We present the results of the MMU-RAGent competition. The competition tasked participants with building systems capable of answering real-world queries drawn from MS MARCO Web Search and Chatbot Arena conversations, as well as queries collected live through the RAG-Arena evaluation platform. To support large-scale retrieval, participants were given API access to the English subsets of ClueWeb22-B and ClueWeb22-A (87M and 800M documents) and AWS-hosted infrastructure for system deployment and evaluation. Submissions were evaluated using a combination of human Likert-scale ratings, live preference judgments collected via RAG-Arena, LLM-as-a-Judge evaluation, and automatic metrics. The competition allowed both open-source and closed-source approaches, including systems leveraging proprietary search APIs or models, with separate leaderboard categories to ensure transparent comparison. Participating systems explored a wide range of RAG architectures, including hierarchical retrieval pipelines, structured document parsing, multi-stage reranking, and iterative generation with citation verification. The competition provides a new benchmark for studying retrieval-augmented generation under realistic deployment conditions and highlights emerging system design patterns for scalable and user-aligned RAG.