LitReview Arena: Evaluating Literature Review Agents with Battle-style Peer Review Platform

Ruotong Zhao, Zhiyu Chen, Xurui Liu, Haidong Xue, Dong Liang, Jigao Fu, Wu Yanbiao, Yuanyi Zhen, Fengli Xu, Yong Li
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:161241-161260, 2026.

Abstract

Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspects of research utility depend on expert judgment rather than reference-overlap metrics. We introduce LitReview Arena, a battle-style evaluation platform with a structured protocol tailored to literature review quality: domain experts with AI paper-writing experience compare anonymized drafts, are matched to topics within their expertise, and provide dimension-wise outcomes over five literature-review-specific criteria. From this protocol, we collect 3k expert judgments, each containing five dimension-wise outcomes, and show that even the strongest current systems win only 23.0% of decisive matches against human drafts on overall utility, while agentic LLMs such as Sonar Deep Research substantially outperform base language models by over 60%. We further find that existing LLM-as-a-judge methods are substantially misaligned with human experts (Spearman’s $\rho \approx 0.467$), especially on synthesis-heavy criteria such as paper structure and research suggestions. Using the collected preference data, we provide an expert-calibrated evaluator, LitJudge, which improves alignment to $\rho \approx 0.78$, comparable to inter-expert consistency; code and data are publicly available at https://github.com/VanellopeAsher/LitReview-Arena.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-zhao26e, title = {{L}it{R}eview Arena: Evaluating Literature Review Agents with Battle-style Peer Review Platform}, author = {Zhao, Ruotong and Chen, Zhiyu and Liu, Xurui and Xue, Haidong and Liang, Dong and Fu, Jigao and Yanbiao, Wu and Zhen, Yuanyi and Xu, Fengli and Li, Yong}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {161241--161260}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhao26e/zhao26e.pdf}, url = {https://proceedings.mlr.press/v306/zhao26e.html}, abstract = {Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspects of research utility depend on expert judgment rather than reference-overlap metrics. We introduce LitReview Arena, a battle-style evaluation platform with a structured protocol tailored to literature review quality: domain experts with AI paper-writing experience compare anonymized drafts, are matched to topics within their expertise, and provide dimension-wise outcomes over five literature-review-specific criteria. From this protocol, we collect 3k expert judgments, each containing five dimension-wise outcomes, and show that even the strongest current systems win only 23.0% of decisive matches against human drafts on overall utility, while agentic LLMs such as Sonar Deep Research substantially outperform base language models by over 60%. We further find that existing LLM-as-a-judge methods are substantially misaligned with human experts (Spearman’s $\rho \approx 0.467$), especially on synthesis-heavy criteria such as paper structure and research suggestions. Using the collected preference data, we provide an expert-calibrated evaluator, LitJudge, which improves alignment to $\rho \approx 0.78$, comparable to inter-expert consistency; code and data are publicly available at https://github.com/VanellopeAsher/LitReview-Arena.} }
Endnote
%0 Conference Paper %T LitReview Arena: Evaluating Literature Review Agents with Battle-style Peer Review Platform %A Ruotong Zhao %A Zhiyu Chen %A Xurui Liu %A Haidong Xue %A Dong Liang %A Jigao Fu %A Wu Yanbiao %A Yuanyi Zhen %A Fengli Xu %A Yong Li %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhao26e %I PMLR %P 161241--161260 %U https://proceedings.mlr.press/v306/zhao26e.html %V 306 %X Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspects of research utility depend on expert judgment rather than reference-overlap metrics. We introduce LitReview Arena, a battle-style evaluation platform with a structured protocol tailored to literature review quality: domain experts with AI paper-writing experience compare anonymized drafts, are matched to topics within their expertise, and provide dimension-wise outcomes over five literature-review-specific criteria. From this protocol, we collect 3k expert judgments, each containing five dimension-wise outcomes, and show that even the strongest current systems win only 23.0% of decisive matches against human drafts on overall utility, while agentic LLMs such as Sonar Deep Research substantially outperform base language models by over 60%. We further find that existing LLM-as-a-judge methods are substantially misaligned with human experts (Spearman’s $\rho \approx 0.467$), especially on synthesis-heavy criteria such as paper structure and research suggestions. Using the collected preference data, we provide an expert-calibrated evaluator, LitJudge, which improves alignment to $\rho \approx 0.78$, comparable to inter-expert consistency; code and data are publicly available at https://github.com/VanellopeAsher/LitReview-Arena.
APA
Zhao, R., Chen, Z., Liu, X., Xue, H., Liang, D., Fu, J., Yanbiao, W., Zhen, Y., Xu, F. & Li, Y.. (2026). LitReview Arena: Evaluating Literature Review Agents with Battle-style Peer Review Platform. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:161241-161260 Available from https://proceedings.mlr.press/v306/zhao26e.html.

Related Material