Elamur Research⌕
☰
home / papers / 2025 / Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025

Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025

Paper · 2025 · Nitya Thakkar, Mert Yuksekgonul, Jake Silberg et al.

Summary

Submissions to AI conferences are growing fast. ICLR grew 47% in 2024 and 61% in 2025. This strains reviewers and leaves authors facing short, unsupported criticism. To address this, the authors built Review Feedback Agent, a pipeline of five Claude Sonnet 3.5 instances: two Actors, an Aggregator, a Critic and a Formatter. It checks each review comment for three problems: vague critiques, questions already answered in the paper, and unprofessional wording. LLM-driven reliability tests serve as guardrails, and any feedback failing a test is regenerated before posting.

The system ran inside OpenReview as a randomized controlled trial. Papers were split into arms in which none, half, or all reviews got feedback, and more than 20,000 reviews received suggestions. Roughly 27% of recipients revised their reviews. Blinded researchers rated the revised versions clearer and more informative, and rebuttal exchanges became longer. The work gives causal, large-scale evidence that carefully filtered LLM feedback can raise peer-review quality while reviewers keep full control.

Key points

  • The trial ran at ICLR 2025 (11,603 submissions) from mid-October to November 12, 2024, using three equal arms. Feedback reached about half of all reviews and was posted about one hour after initial submission. Only the reviewer and program chairs could see it.
  • 26.6% of reviewers who got feedback updated their reviews, incorporating 12,222 agent suggestions in total. Updated reviews grew by an average of 80 words.
  • Reviewers selected for feedback wrote longer replies during rebuttal and prompted longer author responses. They also changed their scores after rebuttal more often.
  • Safeguards included optional use of the feedback, no personal data passed to the model, and an IRB judgment of low risk.
  • The code is released as open source at github.com/zou-group/review_feedback_agent.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 17,959 worksDecision Sciences · 306 worksInformation Systems and Management · 96 worksHistory and Philosophy of Science · 83 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick and James Zou. 2025. Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025. arXiv:2504.09737.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2504.09737 · arXiv 2504.09737

@misc{thakkar2025can,
  title = {{Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025}},
  author = {Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick and James Zou},
  year = {2025},
  doi = {10.48550/arxiv.2504.09737},
  eprint = {2504.09737},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.09737}
}

References

45 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bruce Alberts, Brooks Hanson, and Katrina L. Kelner. Editorial: Reviewing peer review. Science, 321(5885):15–15, 2008. find
  2. Jacalyn Kelly, Tara Sadeghieh, and Khosrow Adeli. Peer review in scientific publications: benefits, critiques, & a survival guide. Ejifcc, 25(3):227, 2014. find
  3. Publons. Global state of peer review 2018, 2018. find
  4. Ariful Azad and Afeefa Banu. Publication trends in artificial intelligence conferences: The rise of super prolific authors, 2024. find
  5. Alison McCook. Is peer review broken? submissions are up, reviewers are overtaxed, and authors are lodging complaint after complaint about the process at top-tier journals. what’s wrong with peer review?, 2006. find
  6. ICLR. Iclr 2024 press release, 2024. find
  7. Anna Rogers and Isabelle Augenstein. What can we do to improve peer review in NLP? In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1256–1262, Online, November 2020. Association for Computational Linguistics. find
  8. Anna Rogers, Marzena Karpinska, Jordan Boyd-Graber, and Naoaki Okazaki. Program chairs’ report on peer review at acl 2023. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages xl–lxxv, 2023. find
  9. Martijn Arns. Open access is tiring out peer reviewers. Nature, 515:467, 2014. find
  10. Corinna Cortes and Neil D. Lawrence. Inconsistency in conference peer review: Revisiting the 2014 neurips experiment, 2021. find
  11. Anthropic. Claude 3.5 sonnet, 2024. find
  12. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8):AIoa2400196, 2024.
  13. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic ”differentiation” via text, 2024. — TextGrad: Automatic "Differentiation" via Text
  14. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-… find
  15. Mohammad Hosseini and Serge P J M Horbach. Fighting reviewer fatigue or amplifying bias? considerations and recommendations for use of chatgpt and other large language models in scholarly peer review. Research Integrity and Peer Review, 2023. find
  16. Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, and James Y. Zou. Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews, 2024.
  17. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023.
  18. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
  19. ICML 2023 program committee. Icml 2023 reviewer tutorial, 2023. find
  20. ICML 2022 Program Chairs. How to be a good reviewer? reviewer tutorial for icml 2022, 2022. find
  21. ACL PC Chairs. Last minute reviewing advice, 2017. find
  22. Matias Valdenegro. Lxcv @ cvpr 2021 reviewer mentoring program: And how to write good reviews, 2021. find
  23. Isabelle Augenstein Anna Rogers. Arr reviewer guidelines, 2021. find
  24. Nyssa J Silbiger and Amber D Stubler. Unprofessional peer reviews disproportionately harm underrepresented groups in stem. PeerJ, 7:e8247, 2019. find
  25. Mathieu Fenniak, Matthew Stamy, pubpub zz, Martin Thoma, Matthew Peveler, exiledkingcc, and pypdf Contributors. The pypdf library, 2024. find
  26. Marco Tulio Ribeiro and Scott Lundberg. Testing language models (and prompts) like we test software, 2023. find
  27. Alexander Goldberg, Ihsan Ullah, Thanh Gia Hieu Khuong, Benedictus Kent Rachmat, Zhen Xu, Isabelle Guyon, and Nihar B. Shah. Usefulness of llms as an author checklist assistant for scientific papers: Neurips’24 experiment, 2024. — Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment
  28. Alex Tamkin, Miles McCain, Kunal Handa, Esin Durmus, Liane Lovitt, Ankur Rathi, Saffron Huang, Alfred Mountfield, Jerry Hong, Stuart Ritchie, Michael Stern, Brian Clarke, Landon Goldberg, Theodore R. Sumers, Jared Mueller, William McEachen, Wes Mitchell, Shan Carter, Jack Clark, Jared Kaplan, and De… find
  29. Ryan Liu and Nihar B. Shah. Reviewergpt? an exploratory study on using large language models for paper reviewing, 2023.
  30. Som Biswas, Dushyant Dobaria, and Harris L. Cohen. Chatgpt and the future of journal reviews: A feasibility study. The Yale Journal of Biology and Medicine, 96(3):415–420, 2023. find
  31. Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, Diyi Yang, Christopher Potts, Christopher D Manning, and James Y. Zou. Mapping the increasing use of llms in scientific papers, 2024.
  32. Nihar B. Shah. Challenges, experiments, and computational solutions in peer review. Commun. ACM, 65(6):76–87, May 2022. find
  33. Simon Price and Peter A. Flach. Computational support for academic peer review: a perspective from artificial intelligence. Commun. ACM, 60(3):70–79, February 2017. find
  34. Atreyi Kankanhalli. Peer review in the age of generative ai. Journal of the Association for Information Systems, 25(1), 2024. find
  35. Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie Névéol, Danish Pruthi, Lizhen Qu, Roy Schwartz, Noah A. Smith, Thamar Solorio, Jingyan Wan… — What Can Natural Language Processing Do for Peer Review?
  36. Tiffany I Leung, Taiane de Azevedo Cardoso, Amaryllis Mavragani, and Gunther Eysenbach. Best practices for using ai tools as an author, peer reviewer, or editor. J Med Internet Res, 25:e51584, Aug 2023. find
  37. Alessandro Checco, Lorenzo Bracciale, Pierpaolo Loreti, Stephen Pinfield, and Giuseppe Bianchi. Ai-assisted peer review. Humanities and Social Sciences Communications, 2021. find
  38. Kayvan Kousha and Mike Thelwall. Artificial intelligence to support publishing and peer review: A summary and review. Learned Publishing, 37(1):4–12, 2024. find
  39. Xiaotian Su, Thiemo Wambsganss, Roman Rietsche, Seyed Parsa Neshaei, and Tanja Käser. Re-viewriter: AI-generated instructions for peer review writing. In Ekaterina Kochmar, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Nitin Madnani, Anaı̈s Tack, Victoria Yaneva, Zheng Yuan, and Torsten Zes… find
  40. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. Marg: Multi-agent review generation for scientific papers, 2024.
  41. Alexander Goldberg, Ivan Stelmakh, Kyunghyun Cho, Alice Oh, Alekh Agarwal, Danielle Belgrave, and Nihar B. Shah. Peer reviews of peer reviews: A randomized controlled trial and other experiments, 2024.
  42. Man Luo, Bradley Peterson, Rafael Gan, Hari Ramalingame, Navya Gangrade, Ariadne Dimarogona, Imon Banerjee, and Phillip Howard. Benchmark on peer review toxic detection: A challenging task with a new dataset, 2025. find
  43. Jon Saad-Falcon, Rajan Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri. Lmunit: Fine-grained evaluation with natural language unit tests, 2024. find
  44. Archiki Prasad, Elias Stengel-Eskin, Justin Chih-Yao Chen, Zaid Khan, and Mohit Bansal. Learning to generate unit tests for automated debugging, 2025. find
  45. Laurent Charlin, Richard S Zemel, and Craig Boutilier. A framework for optimizing paper matching. In UAI, volume 11, pages 86–95, 2011. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thakkar-2025-llm-feedback-enhance-review/