LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake

Haonan Wang, Jiaxiang Liu, Yurong Liu, Austin Senna Wijaya, Tianle Zhou, Eden Wu, Yijia Chen, Wanting You, Reya Vir, Daniela Pinto Veizaga, Grace Fan, Yusen Zhang, Juliana Freire, Eugene Wu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:128436-128463, 2026.

Abstract

Recent large language models (LLMs) have shown rapid progress on reading-based question answering (QA), where the evidence is explicitly provided or trivially retrievable. In contrast, real-world questions are often not paired with accurate evidence documents. The useful evidence resides in a massive collection of data lakes, necessitating searching as a prerequisite for answering. However, there is a lack of a comprehensive benchmark that requires searching and reasoning over a large collection of data lakes. To this end, we introduce LakeQA, a comprehensive benchmark for search-centric question answering over data lakes that jointly emphasizes searching and reasoning capabilities. LakeQA is built on a heterogeneous collection of 9.5 TB text resources from Wikipedia and open-source government data, spanning structured and unstructured data. To ensure the quality of LakeQA’s tasks, each sample is annotated by at least one Ph.D level expert. Each task requires long-horizon multi-hop reasoning with implicit intermediate steps: agents need to discover the correct document(s) and then compose evidence across sources to produce the answer. Intensive experiment results on seven frontier LLMs have demonstrated that LakeQA is challenging. For instance, GPT-5.2 only obtains an exact matching score of 14.73% on LakeQA. Overall LakeQA provides a realistic testbed for developing LLM agents that can both find and analyze data in modern data lakes.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-wang26et, title = {{L}ake{QA}: An Exploratory {QA} Benchmark over a Million-Scale Data Lake}, author = {Wang, Haonan and Liu, Jiaxiang and Liu, Yurong and Wijaya, Austin Senna and Zhou, Tianle and Wu, Eden and Chen, Yijia and You, Wanting and Vir, Reya and Veizaga, Daniela Pinto and Fan, Grace and Zhang, Yusen and Freire, Juliana and Wu, Eugene}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {128436--128463}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wang26et/wang26et.pdf}, url = {https://proceedings.mlr.press/v306/wang26et.html}, abstract = {Recent large language models (LLMs) have shown rapid progress on reading-based question answering (QA), where the evidence is explicitly provided or trivially retrievable. In contrast, real-world questions are often not paired with accurate evidence documents. The useful evidence resides in a massive collection of data lakes, necessitating searching as a prerequisite for answering. However, there is a lack of a comprehensive benchmark that requires searching and reasoning over a large collection of data lakes. To this end, we introduce LakeQA, a comprehensive benchmark for search-centric question answering over data lakes that jointly emphasizes searching and reasoning capabilities. LakeQA is built on a heterogeneous collection of 9.5 TB text resources from Wikipedia and open-source government data, spanning structured and unstructured data. To ensure the quality of LakeQA’s tasks, each sample is annotated by at least one Ph.D level expert. Each task requires long-horizon multi-hop reasoning with implicit intermediate steps: agents need to discover the correct document(s) and then compose evidence across sources to produce the answer. Intensive experiment results on seven frontier LLMs have demonstrated that LakeQA is challenging. For instance, GPT-5.2 only obtains an exact matching score of 14.73% on LakeQA. Overall LakeQA provides a realistic testbed for developing LLM agents that can both find and analyze data in modern data lakes.} }
Endnote
%0 Conference Paper %T LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake %A Haonan Wang %A Jiaxiang Liu %A Yurong Liu %A Austin Senna Wijaya %A Tianle Zhou %A Eden Wu %A Yijia Chen %A Wanting You %A Reya Vir %A Daniela Pinto Veizaga %A Grace Fan %A Yusen Zhang %A Juliana Freire %A Eugene Wu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wang26et %I PMLR %P 128436--128463 %U https://proceedings.mlr.press/v306/wang26et.html %V 306 %X Recent large language models (LLMs) have shown rapid progress on reading-based question answering (QA), where the evidence is explicitly provided or trivially retrievable. In contrast, real-world questions are often not paired with accurate evidence documents. The useful evidence resides in a massive collection of data lakes, necessitating searching as a prerequisite for answering. However, there is a lack of a comprehensive benchmark that requires searching and reasoning over a large collection of data lakes. To this end, we introduce LakeQA, a comprehensive benchmark for search-centric question answering over data lakes that jointly emphasizes searching and reasoning capabilities. LakeQA is built on a heterogeneous collection of 9.5 TB text resources from Wikipedia and open-source government data, spanning structured and unstructured data. To ensure the quality of LakeQA’s tasks, each sample is annotated by at least one Ph.D level expert. Each task requires long-horizon multi-hop reasoning with implicit intermediate steps: agents need to discover the correct document(s) and then compose evidence across sources to produce the answer. Intensive experiment results on seven frontier LLMs have demonstrated that LakeQA is challenging. For instance, GPT-5.2 only obtains an exact matching score of 14.73% on LakeQA. Overall LakeQA provides a realistic testbed for developing LLM agents that can both find and analyze data in modern data lakes.
APA
Wang, H., Liu, J., Liu, Y., Wijaya, A.S., Zhou, T., Wu, E., Chen, Y., You, W., Vir, R., Veizaga, D.P., Fan, G., Zhang, Y., Freire, J. & Wu, E.. (2026). LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:128436-128463 Available from https://proceedings.mlr.press/v306/wang26et.html.

Related Material