LiveNewsBench: Evaluating Web Search Agents with Freshly Curated News

Yunfan Zhang, Kathleen Mckeown, Smaranda Muresan
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:157763-157786, 2026.

Abstract

Large Language Models (LLMs) with agentic web search capabilities show strong potential for tasks requiring real-time information access and complex fact retrieval, yet evaluating such systems remains challenging. We introduce LiveNewsBench, a rigorous and regularly updated benchmark designed to assess the agentic web search abilities of LLMs. LiveNewsBench automatically generates fresh question-answer pairs from recent news articles, ensuring that questions require information beyond an LLM’s training data and enabling clear separation between internal knowledge and search capability. The benchmark features intentionally difficult questions requiring multi-hop search queries, page visits, and reasoning, making it well-suited for evaluating agentic search behavior. Our automated data curation and question generation pipeline enables frequent benchmark updates and supports construction of a large-scale training dataset for agentic web search models, addressing the scarcity of such data in the research community. To ensure reliable evaluation, we include a subset of human-verified samples in the test set. We evaluate a broad range of systems using LiveNewsBench, including commercial and open-weight LLMs as well as LLM-based web search APIs. The leaderboard, datasets, and code are publicly available at livenewsbench.com.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-zhang26ej, title = {{L}ive{N}ews{B}ench: Evaluating Web Search Agents with Freshly Curated News}, author = {Zhang, Yunfan and Mckeown, Kathleen and Muresan, Smaranda}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {157763--157786}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhang26ej/zhang26ej.pdf}, url = {https://proceedings.mlr.press/v306/zhang26ej.html}, abstract = {Large Language Models (LLMs) with agentic web search capabilities show strong potential for tasks requiring real-time information access and complex fact retrieval, yet evaluating such systems remains challenging. We introduce LiveNewsBench, a rigorous and regularly updated benchmark designed to assess the agentic web search abilities of LLMs. LiveNewsBench automatically generates fresh question-answer pairs from recent news articles, ensuring that questions require information beyond an LLM’s training data and enabling clear separation between internal knowledge and search capability. The benchmark features intentionally difficult questions requiring multi-hop search queries, page visits, and reasoning, making it well-suited for evaluating agentic search behavior. Our automated data curation and question generation pipeline enables frequent benchmark updates and supports construction of a large-scale training dataset for agentic web search models, addressing the scarcity of such data in the research community. To ensure reliable evaluation, we include a subset of human-verified samples in the test set. We evaluate a broad range of systems using LiveNewsBench, including commercial and open-weight LLMs as well as LLM-based web search APIs. The leaderboard, datasets, and code are publicly available at livenewsbench.com.} }
Endnote
%0 Conference Paper %T LiveNewsBench: Evaluating Web Search Agents with Freshly Curated News %A Yunfan Zhang %A Kathleen Mckeown %A Smaranda Muresan %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhang26ej %I PMLR %P 157763--157786 %U https://proceedings.mlr.press/v306/zhang26ej.html %V 306 %X Large Language Models (LLMs) with agentic web search capabilities show strong potential for tasks requiring real-time information access and complex fact retrieval, yet evaluating such systems remains challenging. We introduce LiveNewsBench, a rigorous and regularly updated benchmark designed to assess the agentic web search abilities of LLMs. LiveNewsBench automatically generates fresh question-answer pairs from recent news articles, ensuring that questions require information beyond an LLM’s training data and enabling clear separation between internal knowledge and search capability. The benchmark features intentionally difficult questions requiring multi-hop search queries, page visits, and reasoning, making it well-suited for evaluating agentic search behavior. Our automated data curation and question generation pipeline enables frequent benchmark updates and supports construction of a large-scale training dataset for agentic web search models, addressing the scarcity of such data in the research community. To ensure reliable evaluation, we include a subset of human-verified samples in the test set. We evaluate a broad range of systems using LiveNewsBench, including commercial and open-weight LLMs as well as LLM-based web search APIs. The leaderboard, datasets, and code are publicly available at livenewsbench.com.
APA
Zhang, Y., Mckeown, K. & Muresan, S.. (2026). LiveNewsBench: Evaluating Web Search Agents with Freshly Curated News. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:157763-157786 Available from https://proceedings.mlr.press/v306/zhang26ej.html.

Related Material