Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review

Kai Lu, Shixiong Xu, Jinqiu Li, Kun Ding, Gaofeng Meng
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:40803-40830, 2025.

Abstract

Feedback from peer review is essential to improve the quality of scientific articles. However, at present, many manuscripts do not receive sufficient external feedback for refinement before or during submission. Therefore, a system capable of providing detailed and professional feedback is crucial for enhancing research efficiency. In this paper, we have compiled the largest dataset of paper reviews to date by collecting historical open-access papers and their corresponding review comments and standardizing them using LLM. We then developed a multi-agent system that mimics real human review processes, based on LLMs. This system, named Agent Reviewers, includes the innovative introduction of multimodal reviewers to provide feedback on the visual elements of papers. Additionally, a shared memory pool that stores historical papers’ metadata is preserved, which supplies reviewer agents with background knowledge from different fields. Our system is evaluated using ICLR 2024 papers and achieves superior performance compared to existing AI-based review systems. Comprehensive ablation studies further demonstrate the effectiveness of each module and agent in this system.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-lu25p, title = {Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review}, author = {Lu, Kai and Xu, Shixiong and Li, Jinqiu and Ding, Kun and Meng, Gaofeng}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {40803--40830}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/lu25p/lu25p.pdf}, url = {https://proceedings.mlr.press/v267/lu25p.html}, abstract = {Feedback from peer review is essential to improve the quality of scientific articles. However, at present, many manuscripts do not receive sufficient external feedback for refinement before or during submission. Therefore, a system capable of providing detailed and professional feedback is crucial for enhancing research efficiency. In this paper, we have compiled the largest dataset of paper reviews to date by collecting historical open-access papers and their corresponding review comments and standardizing them using LLM. We then developed a multi-agent system that mimics real human review processes, based on LLMs. This system, named Agent Reviewers, includes the innovative introduction of multimodal reviewers to provide feedback on the visual elements of papers. Additionally, a shared memory pool that stores historical papers’ metadata is preserved, which supplies reviewer agents with background knowledge from different fields. Our system is evaluated using ICLR 2024 papers and achieves superior performance compared to existing AI-based review systems. Comprehensive ablation studies further demonstrate the effectiveness of each module and agent in this system.} }
Endnote
%0 Conference Paper %T Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review %A Kai Lu %A Shixiong Xu %A Jinqiu Li %A Kun Ding %A Gaofeng Meng %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-lu25p %I PMLR %P 40803--40830 %U https://proceedings.mlr.press/v267/lu25p.html %V 267 %X Feedback from peer review is essential to improve the quality of scientific articles. However, at present, many manuscripts do not receive sufficient external feedback for refinement before or during submission. Therefore, a system capable of providing detailed and professional feedback is crucial for enhancing research efficiency. In this paper, we have compiled the largest dataset of paper reviews to date by collecting historical open-access papers and their corresponding review comments and standardizing them using LLM. We then developed a multi-agent system that mimics real human review processes, based on LLMs. This system, named Agent Reviewers, includes the innovative introduction of multimodal reviewers to provide feedback on the visual elements of papers. Additionally, a shared memory pool that stores historical papers’ metadata is preserved, which supplies reviewer agents with background knowledge from different fields. Our system is evaluated using ICLR 2024 papers and achieves superior performance compared to existing AI-based review systems. Comprehensive ablation studies further demonstrate the effectiveness of each module and agent in this system.
APA
Lu, K., Xu, S., Li, J., Ding, K. & Meng, G.. (2025). Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:40803-40830 Available from https://proceedings.mlr.press/v267/lu25p.html.

Related Material