Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation

Shenglai Zeng, Jiankun Zhang, Kai Guo, Xinnan Dai, Hui Liu, Jiliang Tang, Yi Chang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:153711-153743, 2026.

Abstract

Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a key paradigm for grounding MLLMs with external knowledge. While query pre-processing (e.g., rewriting) is standard in text-based RAG, existing MRAG pipelines predominantly treat visual inputs as static and immutable, implicitly assuming they are noise-free. However, real-world visual queries are often “imperfect”—suffering from geometric distortions, quality degradation, or semantic ambiguity—leading to catastrophic retrieval failures. To address this gap, we propose V-QPP-Bench, the first comprehensive benchmark dedicated to Visual Query Pre-processing (V-QPP). We formulate V-QPP as an agentic decision-making task where MLLMs must autonomously diagnose imperfections and deploy perceptual tools to refine queries. Our extensive evaluation across 46,700 imperfect queries and diverse MRAG paradigms reveals three critical insights: (1) Vulnerability—visual imperfections severely degrade both retrieval recall and end-to-end MRAG performance; (2) Restoration Potential & Bottleneck—while oracle preprocessing recovers near-perfect performance, off-the-shelf MLLMs struggle with tool selection and parameter prediction without specialized training; and (3) Training Enhancement—supervised fine-tuning enables compact models to achieve comparable or superior performance to larger proprietary models, demonstrating the benchmark’s value for developing robust MRAG systems The code is available at https://github.com/phycholosogy/VQQP_Bench

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-zeng26w, title = {Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation}, author = {Zeng, Shenglai and Zhang, Jiankun and Guo, Kai and Dai, Xinnan and Liu, Hui and Tang, Jiliang and Chang, Yi}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {153711--153743}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zeng26w/zeng26w.pdf}, url = {https://proceedings.mlr.press/v306/zeng26w.html}, abstract = {Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a key paradigm for grounding MLLMs with external knowledge. While query pre-processing (e.g., rewriting) is standard in text-based RAG, existing MRAG pipelines predominantly treat visual inputs as static and immutable, implicitly assuming they are noise-free. However, real-world visual queries are often “imperfect”—suffering from geometric distortions, quality degradation, or semantic ambiguity—leading to catastrophic retrieval failures. To address this gap, we propose V-QPP-Bench, the first comprehensive benchmark dedicated to Visual Query Pre-processing (V-QPP). We formulate V-QPP as an agentic decision-making task where MLLMs must autonomously diagnose imperfections and deploy perceptual tools to refine queries. Our extensive evaluation across 46,700 imperfect queries and diverse MRAG paradigms reveals three critical insights: (1) Vulnerability—visual imperfections severely degrade both retrieval recall and end-to-end MRAG performance; (2) Restoration Potential & Bottleneck—while oracle preprocessing recovers near-perfect performance, off-the-shelf MLLMs struggle with tool selection and parameter prediction without specialized training; and (3) Training Enhancement—supervised fine-tuning enables compact models to achieve comparable or superior performance to larger proprietary models, demonstrating the benchmark’s value for developing robust MRAG systems The code is available at https://github.com/phycholosogy/VQQP_Bench} }
Endnote
%0 Conference Paper %T Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation %A Shenglai Zeng %A Jiankun Zhang %A Kai Guo %A Xinnan Dai %A Hui Liu %A Jiliang Tang %A Yi Chang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zeng26w %I PMLR %P 153711--153743 %U https://proceedings.mlr.press/v306/zeng26w.html %V 306 %X Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a key paradigm for grounding MLLMs with external knowledge. While query pre-processing (e.g., rewriting) is standard in text-based RAG, existing MRAG pipelines predominantly treat visual inputs as static and immutable, implicitly assuming they are noise-free. However, real-world visual queries are often “imperfect”—suffering from geometric distortions, quality degradation, or semantic ambiguity—leading to catastrophic retrieval failures. To address this gap, we propose V-QPP-Bench, the first comprehensive benchmark dedicated to Visual Query Pre-processing (V-QPP). We formulate V-QPP as an agentic decision-making task where MLLMs must autonomously diagnose imperfections and deploy perceptual tools to refine queries. Our extensive evaluation across 46,700 imperfect queries and diverse MRAG paradigms reveals three critical insights: (1) Vulnerability—visual imperfections severely degrade both retrieval recall and end-to-end MRAG performance; (2) Restoration Potential & Bottleneck—while oracle preprocessing recovers near-perfect performance, off-the-shelf MLLMs struggle with tool selection and parameter prediction without specialized training; and (3) Training Enhancement—supervised fine-tuning enables compact models to achieve comparable or superior performance to larger proprietary models, demonstrating the benchmark’s value for developing robust MRAG systems The code is available at https://github.com/phycholosogy/VQQP_Bench
APA
Zeng, S., Zhang, J., Guo, K., Dai, X., Liu, H., Tang, J. & Chang, Y.. (2026). Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:153711-153743 Available from https://proceedings.mlr.press/v306/zeng26w.html.

Related Material