Elamur Research⌕
☰
home / papers / 2026 / Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future

Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future

Paper · 2026 · Sihong Wu, Owen Jiang, Yilun Zhao et al.

Summary

This survey asks how far large language models can support or automate scholarly peer review. It treats review as a full pipeline (reviewing, rebuttal, meta-review, decision, and revision) rather than a single text-generation step. The authors build a taxonomy across three areas:

  • Review generation, split into five paradigms: pre-LLM foundation work, fine-tuning, agent-based systems, reinforcement learning, and enhancement techniques such as retrieval, iterative refinement, and structure or style control.
  • After-review tasks: rebuttal writing, meta-review synthesis, and manuscript revision driven by reviewer comments.
  • Evaluation, grouped as human-centric, reference-based, LLM-as-judge, and aspect-oriented.

They also catalog the datasets and metrics in use. The survey concludes that LLMs now produce fluent, well-formatted reviews but still struggle with novelty judgment and content-level quality assessment. Benchmarks are concentrated in NLP and ML venues. Ethical risks such as bias are significant. The work matters as a practical map for anyone building or assessing AI-assisted reviewing tools. The authors stress that such tools should assist human reviewers, not replace them.

Key points

  • Fine-tuning: OpenReviewer fine-tunes Llama-8B on about 79,000 curated expert reviews, and REVIEWER2 draws on roughly 27k papers with 99k reviews. Supervised fine-tuning on ReviewMT beats zero-shot prompting on review and decision hit rates.
  • Agents and RL: Agent-based systems include MARG, SWIF²T, ReviewAgents, DeepReview, MAMORX, and AgentReview. RL approaches include REMOR (GRPO), CycleResearcher (SimPO), ReviewRL, and REM-CTX.
  • Datasets: The foundational resources are PeerRead, NLpeer, and MOPRD, with MOPRD being multidisciplinary. ReviewCritique and SubstanReview offer fine-grained labels for content-aware evaluation.
  • Metrics: Evaluation still relies heavily on coarse overlap metrics such as BLEU, ROUGE, and BertScore. Novelty assessment remains a weak spot, with SchNovel and NovBench as dedicated benchmarks.
  • Ethics: According to one cited corpus study, LLMs substantially modified an estimated 6.5–16.9% of the reviewing text submitted to leading AI conferences. Studies also document affiliation bias in LLM reviewers. The authors call for mandatory disclosure of LLM use.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States201 works · 100,180 citations🎓Yale UniversityNew Haven · United States95 works · 12,765 citations🏢Tata Consultancy ServicesPune · India8 works · 35 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksHistory and Philosophy of Science · 53 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Sihong Wu, Owen Jiang, Yilun Zhao, Tiansheng Hu, Yiling Ma, Kaiyan Zhang, Manasi Patwardhan and Arman Cohan. 2026. Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1504

@inproceedings{wu2026can,
  title = {{Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future}},
  author = {Sihong Wu, Owen Jiang, Yilun Zhao, Tiansheng Hu, Yiling Ma, Kaiyan Zhang, Manasi Patwardhan and Arman Cohan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1504},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1504}
}

References

160 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2008–2025. Grobid. https://github.com/ kermitt2/grobid. 2017–2025. Science parse. https://github. com/allenai/science-parse. 2023–2025. Marker. https://github.com/ datalab-to/marker. ↗
  2. Osama Mohammed Afzal, Preslav Nakov, Tom Hope, and Iryna Gurevych. 2025. Beyond "not novel enough": Enriching scholarly critique with llm-assisted feedback. find
  3. Mistral AI. 2023. Mistral-7b-instruct-v0.2. https://huggingface.co/mistralai/ Accessed: Mistral-7B-Instruct-v0.2. 2025-07-28. ↗
  4. Mistral AI. 2024. Mistral large. https://mistral.ai/news/mistral-large/. Accessed: 2025-07-28. ↗
  5. AI@Meta. 2024. Llama 3 model card. find
  6. Miltiadis Allamanis, Sheena Panthaplackel, and Pengcheng Yin. 2024. Unsupervised evaluation of code llms with round-trip correctness. find
  7. Hiroki Asano, Tomoya Mizumoto, and Kentaro Inui. 2017. Reference-based metrics can be replaced with reference-less metrics in evaluating grammatical error correction systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 34…
  8. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — Qwen Technical Report
  9. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… find
  10. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. find
  11. Prabhat Kumar Bharti, Shashi Ranjan, Tirthankar Ghosal, Mayank Agrawal, and Asif Ekbal. 2021. PEERAssist: Leveraging on paper–review interactions to predict peer review decisions. In Proceedings of the 23rd International Conference on Asia-Pacific Digital Libraries (ICADL 2021), Lecture Notes in Com… find
  12. Chaitanya Bhatia, Tribikram Pradhan, and Sukomal Pal. 2020. Metagen: An academic meta-review generation system. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 1653–1656, New York, NY, USA. Association for Computing… find
  13. Rishi Bommasani, Jack Clark Hudson, Ehsan Adeli, Sam Altman, Simran Arora, Sydney von Arx, Michael Bernstein, and et al. 2021. On the opportunities and risks of foundation models.
  14. N. C. Büyükkaramikli, M. P. M. H. Rutten-van Mölken, J. L. Severens, and M. Al. 2019. TECH-VER: A verification checklist to reduce errors in models and improve their credibility. Pharmacoeconomics, 37(11):1391–1408. find
  15. Eric Chamoun, Michael Schlichtkrull, and Andreas Vlachos. 2024. Automated focused feedback generation for scientific writing assistance. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9742–9763, Bangkok, Thailand. Association for Computational Linguistics. find
  16. Yuan Chang, Ziyue Li, Hengyuan Zhang, Yuanbo Kong, Yanru Wu, Zhijiang Guo, and Ngai Wong. 2025. Treereview: A dynamic tree of questions framework for deep and efficient llm-based scientific peer review.
  17. A. Checco, L. Bracciale, P. Loreti, et al. 2021. Ai-assisted peer review. Humanities and Social Sciences Communications, 8:25. find
  18. Qiguang Chen, Mingda Yang, Libo Qin, Jinhao Liu, Zheng Yan, Jiannan Guan, Dengyun Peng, Yiyan Ji, Hanjing Li, Mengkang Hu, Yimeng Zhang, Yihao Liang, Yuhang Zhou, Jiaqi Wang, Zhi Chen, and Wanxiang Che. 2025. Ai4research: A survey of artificial intelligence for scientific research.
  19. Liying Cheng, Lidong Bing, Qian Yu, Wei Lu, and Luo Si. 2020. APE: Argument pair extraction from peer review and rebuttal via multi-task learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7000–7011, Online. Association for Computational… find
  20. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating llms by human preference.
  21. Maitreya Prafulla Chitale, Ketaki Mangesh Shetye, Harshit Gupta, Manav Chaudhary, and Vasudeva Varma. 2025. Autorev: Automatic peer review system for academic research papers. find
  22. Gautam Choudhary, Natwar Modani, and Nitish Maurya. 2022. React: A review comment dataset for actionability (and more). find
  23. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2023. Deep reinforcement learning from human preferences. find
  24. Magic-Doc Contributors. 2024. Magic-doc: A toolkit that converts multiple file types to markdown. https://github.com/InternLM/ magic-doc. ↗
  25. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. Marg: Multi-agent review generation for scientific papers. find
  26. Mike D’Arcy, Alexis Ross, Erin Bransom, Bailey Kuehl, Jonathan Bragg, Tom Hope, and Doug Downey. 2023. ARIES: A corpus of scientific paper edits made in response to peer reviews.
  27. Iddo Drori and Dov Te’eni. 2024. Human-in-the-loop ai reviewing: Feasibility, opportunities, and risks. Journal of the Association for Information Systems, 25(1):98–109. find
  28. J. A. Drozdz and M. R. Ladomery. 2024. The peer review process: Past, present, and future. British Journal of Biomedical Science, 81:12054. find
  29. Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Ranran Haoran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Jiayang Cheng, Zhaowei Wang, Yi… — LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
  30. Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Li Fei-Fei, and Jianfeng Gao. 2024. Agent ai: Surveying the horizons of multimodal interaction. find
  31. Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2022. Yes-yes-yes: Proactive data collection for acl rolling review and beyond. find
  32. Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023a. Nlpeer: A unified resource for the computational study of peer review.
  33. Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023b. Overview of PragTag-2023: Low-resource multi-domain pragmatic tagging of peer reviews. In Proceedings of the 10th Workshop on Argument Mining, pages 187–196, Singapore. Association for Computational Linguistics.
  34. Nils Dycke, Matej Zečević, Ilia Kuznetsov, Beatrix Suess, Kristian Kersting, and Iryna Gurevych. 2025. STRICTA: Structured reasoning in critical text assessment for peer review and beyond. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa… find
  35. Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. 2019. Multi-news: a large-scale multi-document summarization dataset and abstractive hierarchical model.
  36. Abdur Rahman Bin Md Faizullah, Ashok Urlana, and Rahul Mishra. 2024. Limgen: Probing the llms for generating suggestive limitations of research papers. find
  37. Kavita Ganesan. 2018. Rouge 2.0: Updated and improved measures for evaluation of summarization tasks. find
  38. Xian Gao, Jiacheng Ruan, Zongyun Zhang, Jingsheng Gao, Ting Liu, and Yuzhuo Fu. 2025. Reviewagents: Bridging the gap between human and ai-generated paper reviews. find
  39. Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. 2019. Does my rebuttal matter? insights from a major NLP conference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (… find
  40. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024a. Retrieval-augmented generation for large language models: A survey.
  41. Zhaolin Gao, Kianté Brantley, and Thorsten Joachims. 2024b. Reviewer2: Optimizing review generation through prompt generation.
  42. Zhaolin Gao, Kianté Brantley, and Thorsten Joachims. 2024c. Reviewer2: Optimizing review generation through prompt generation.
  43. Madhav Krishan Garg, Chhavi Kirtani, Tejash Prasad, Tanmay Singhal, Murari Mandal, and Dhruv Kumar. 2024. Revieweval: An evaluation framework for ai-generated reviews. arXiv preprint arXiv:2502.11736. find
  44. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. Critic: Large language models can self-correct with tool-interactive critiquing.
  45. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge. find
  46. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2024. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
  47. Yanzhu Guo, Guokan Shang, Virgile Rennard, Michalis Vazirgiannis, and Chloé Clavel. 2023. Automatic analysis of substantiation in scientific peer reviews. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10198–10216. find
  48. Haixia Han, Jiaqing Liang, Jie Shi, Qianyu He, and Yanghua Xiao. 2024. Small language model can self-correct. find
  49. Peixuan Han, Yingjie Yu, Jingjun Xu, and Jiaxuan You. 2026. Drpg (decompose, retrieve, plan, generate): An agentic framework for academic rebuttal. find
  50. Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. 2021. Machine learning with a reject option: A survey. IEEE Transactions on Knowledge and Data Engineering, 33(1):43–54. find
  51. M. Hosseini and S. P. J. M. Horbach. 2023. Fighting reviewer fatigue or amplifying bias? considerations and recommendations for use of chatgpt and other large language models in scholarly peer review. Research Integrity and Peer Review, 8(1):4. find
  52. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
  53. Junjie Huang, Win-bin Huang, Yi Bu, Qi Cao, Huawei Shen, and Xueqi Cheng. 2023. What makes a successful rebuttal in computer science conferences?: A perspective on social interaction. Journal of Informetrics, 17(3):101427. find
  54. Maximilian Idahl and Zahra Ahmadi. 2025. Openreviewer: A specialized large language model for generating critical scientific paper reviews. find
  55. Jinbae Im, Moonki Kim, Hoyeop Lee, Hyunsouk Cho, and Sehee Chung. 2021. Self-supervised multimodal opinion summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: L… find
  56. Chao Jiang, Wei Xu, and Samuel Stevens. 2022. arxivedits: Understanding the human revision process in scientific writing. find
  57. Feng Jiang, Kuang Wang, and Haizhou Li. 2024. Bridging research and readers: A multi-modal automated academic papers interpretation system. find
  58. Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang. 2024a. Agentreview: Exploring peer review dynamics with llm agents. find
  59. Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang. 2024b. AgentReview: Exploring peer review dynamics with LLM agents. find
  60. Leane Jourdan, Florian Boudin, Nicolas Hernandez, and Richard Dufour. 2024. Casimir: A corpus of scientific articles enhanced with multiple author-integrated revisions. find
  61. Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. When can LLMs actually correct their own mistakes? a critical survey of self-correction of LLMs. Transactions of the Association for Computational Linguistics, 12:1417–1440.
  62. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Comp…
  63. Neha Kennard, Tim O’Gorman, Rajarshi Das, Akshay Sharma, Chhandak Bagchi, Matthew Clinton, Pranay Kumar Yelugam, Hamed Zamani, and Andrew McCallum. 2021. Disapere: A dataset for discourse structure in peer review discussions. find
  64. Jyotsana Khatri and Manasi Patwardhan. 2026. Defend: Automated rebuttals for peer review with minimal author guidance. find
  65. Rodney Michael Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Gorney, David W. Graham,… find
  66. Barbara Kitchenham and Stuart Charters. 2007. Guidelines for performing systematic literature reviews in software engineering technical report. Software Engineering Group, EBSE Technical Report, Keele University and Department of Computer Science University of Durham, 2. find
  67. Miao Li, Jianzhong Qi, and Jey Han Lau. 2022. Peer-sum: A peer review dataset for abstractive multi-document summarization. find
  68. Asheesh Kumar, Amrinder Arora, Patrick Derieux, and Sandeep Purao. 2024a. Towards automated meta-review generation via an nlp/ml pipeline in different stages of the scholarly peer review process. International Journal on Digital Libraries, 25(3):295–314. find
  69. Wei Li and Hai Zhuge. 2021. Abstractive multi-document summarization based on semantic link network. IEEE Transactions on Knowledge and Data Engineering, 33(1):43–54. find
  70. Asheesh Kumar, Tirthankar Ghosal, and Asif Ekbal. 2024b. A deep neural architecture for decision-aware meta-review generation. In Proceedings of the 2021 ACM/IEEE Joint Conference on Digital Libraries, JCDL ’21, page 222–225. IEEE Press. find
  71. Sandeep Kumar, Guneet Singh Kohli, Tirthankar Ghosal, and Asif Ekbal. 2024c. Longform multimodal lay summarization of scientific papers: Towards automatically generating science blogs from research articles. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Lang… find
  72. Mayte H Laureano, Hiram Calvo, Tania Alcantara, Omar Garcıa Vazquez, and Marco A Cardoso Moreno. 2025. Predicting Reviewers’ Decisions in Scientific Submissions through Linguistic Analysis. Journal of Scientometric Research, 14(1):331–341. find
  73. Carole J Lee, Cassidy R Sugimoto, Guo Zhang, and Blaise Cronin. 2013. Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1):2–17. find
  74. J Lee, J Lee, and J. J. Yoo. 2025. The role of large language models in the peer-review process: opportunities and challenges for medical journal reviewers and editors. Journal of Educational Evaluation for Health Professions, 22:4. find
  75. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-augmented generation for knowledge-intensive nlp tasks.
  76. Miao Li, Eduard Hovy, and Jey Lau. 2023. Summarizing multiple documents with conversational structure for meta-review generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7089–7112, Singapore. Association for Computational Linguistics. find
  77. Miao Li, Jey Han Lau, and Eduard Hovy. 2024. A sentiment consolidation framework for meta-review generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10158–10177, Bangkok, Thailand. Association for Computational Linguis… find
  78. Pan Li and Alexander Tuzhilin. 2020. Towards controllable and personalized review generation. find
  79. Yinheng Li. 2023. A practical survey on zero-shot prompt design for in-context learning. In Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Processings, RANLP, page 641–647. INCOMA Ltd., Shoumen, BULGARIA. find
  80. Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, and James Y. Zou. 2024. Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews. find
  81. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Smith, Yian Yin, Daniel McFarland, and James Zou. 2023. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. find
  82. Zeyi Liao, Jaylen Jones, Linxi Jiang, Eric Fosler-Lussier, Yu Su, Zhiqiang Lin, and Huan Sun. 2025. Redteam-cua: Realistic adversarial testing of computer-use agents in hybrid web-os environments. find
  83. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  84. Ethan Lin, Zhiyuan Peng, and Yi Fang. 2024. Evaluating and enhancing large language models for novelty assessment in scholarly publications. find
  85. Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. 2022. Moprd: A multidisciplinary open peer review dataset. arXiv preprint arXiv:2212.04972. — [arXiv:2212.04972](https://arxiv.org/abs/2212.04972) find
  86. Yen-Ting Lin and Yun-Nung Chen. 2023. Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models.
  87. Ryan Liu and Nihar B. Shah. 2023. Reviewergpt? an exploratory study on using large language models for paper reviewing. find
  88. Sheng Lu, Ilia Kuznetsov, and Iryna Gurevych. 2025. Identifying aspects in peer reviews. find
  89. Qianli Ma, Chang Guo, Zhiheng Tian, Siyu Wang, Jipeng Xiao, Yuanhao Yue, and Zhipeng Zhang. 2026. Paper2rebuttal: A multi-agent framework for transparent author response assistance.
  90. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. find
  91. Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitsc… find
  92. Panitan Muangkammuen, Fumiyo Fukumoto, Jiyi Li, and Yoshimi Suzuki. 2022. Exploiting labeled and unlabeled data via transformer fine-tuning for peer-review score prediction. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2233–2240. find
  93. Jimin Mun, Chani Jung, Xuhui Zhou, Hyunwoo Kim, and Maarten Sap. 2026. Goodpoint: Learning constructive scientific paper feedback from author responses. find
  94. Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. 2023. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. ACM Computing Surveys, 55(13s):1–4… find
  95. Kun-Peng Ning, Shuo Yang, Yu-Yang Liu, Jia-Yu Yao, Zhen-Hui Liu, Yong-Hong Tian, Yibing Song, and Li Yuan. 2025. Pico: Peer review in llms based on the consistency optimization. find
  96. Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Ba… — AlphaEvolve: A coding agent for scientific and algorithmic discovery
  97. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… find
  98. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA. Association for Computational Linguistics.
  99. Sukannya Purkayastha, Nils Dycke, Anne Lauscher, and Iryna Gurevych. 2026. Decision-making with deliberation: Meta-reviewing as a document-grounded dialogue.
  100. Sukannya Purkayastha, Anne Lauscher, and Iryna Gurevych. 2023a. Exploring jiu-jitsu argumentation for writing peer review rebuttals. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14479–14495, Singapore. Association for Computational Linguistics. find
  101. Sukannya Purkayastha, Anne Lauscher, and Iryna Gurevych. 2023b. Exploring jiu-jitsu argumentation for writing peer review rebuttals. find
  102. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Technical Report. https://cdn.openai.com/better-language-models/ language_models_are_unsupervised_ multitask_learners.pdf. ↗
  103. Abdelrahman Sadallah, Tim Baumgärtner, Iryna Gurevych, and Ted Briscoe. 2025. The good, the bad and the constructive: Automatically measuring peer review’s utility for authors. find
  104. Laurie A. Schintler, Connie L. McNeely, and James Witte. 2023. A critical examination of the ethics of ai-mediated peer review. find
  105. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
  106. Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. 2021. MReD: A meta-review dataset for structure-controllable text generation. find
  107. Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. 2022. MReD: A meta-review dataset for structure-controllable text generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2521–2535, Dublin, Ireland. Association for Computational Linguistics. find
  108. Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, and Juho Kim. 2025. Mind the blind spots: A focus-level evaluation framework for llm reviews.
  109. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation.
  110. Sonish Sivarajkumar, Mark Kelley, Alyssa Samolyk-Mazzanti, Shyam Visweswaran, and Yanshan Wang. 2023. An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing. find
  111. Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Ibarz, and Mohi Khansari. 2020. Rl-cyclegan: Reinforcement learning aware simulation-to-real. find
  112. Purin Sukpanichnant, Anna Rapberger, and Francesca Toni. 2024. Peerarg: Argumentative peer review with llms. arXiv preprint arXiv:2409.16813. find
  113. Clément Reverdy, Sylvie Gibet, and Thibaut Le Naour. 2024. STK LSF: A motion capture dataset in LSF for SignToKids. In Proceedings of the LREC-COLING 2024 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language Resources, pages 315–322, Torino, Italia. ELRA… find
  114. Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J. Andrew Bagnell. 2025. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. find
  115. Zachary Robertson. 2023. Gpt4 is slightly helpful for peer-review assistance: A pilot study. find
  116. Qian Ruan and Iryna Gurevych. 2026. Author-in-the-loop response generation and evaluation: Integrating author expertise and intent in responses to peer review.
  117. Sajratul Y. Rubaiat, Syed N. Sakib, and Hasan M. Jamil. 2025. Mapping the evolution of research contributions using knovo. find
  118. Pawin Taechoyotin and Daniel Acuna. 2025. Remor: Automated peer review generation with llm reasoning and multi-objective reinforcement learning. find
  119. Pawin Taechoyotin and Daniel E. Acuna. 2026. Rem-ctx: Automated peer review via reinforcement learning with auxiliary context. find
  120. Pawin Taechoyotin, Guanchao Wang, Tong Zeng, Bradley Sides, and Daniel Acuna. 2024. MAMORX: Multi-agent multi-modal scientific review generation with external knowledge. In Neurips 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenges. find
  121. Cheng Tan, Dongxin Lyu, Siyuan Li, Zhangyang Gao, Jingxuan Wei, Siqi Ma, Zicheng Liu, and Stan Z. Li. 2024. Peer review as a multi-turn and long-context dialogue with role-based interactions. find
  122. Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. 2025. Ai-researcher: Autonomous scientific innovation. find
  123. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  124. Simone Teufel, Jean Carletta, and Marc Moens. 1999. An annotation scheme for discourse-level argumentation in research articles. In Ninth Conference of the European Chapter of the Association for Computational Linguistics, pages 110–117, Bergen, Norway. Association for Computational Linguistics. find
  125. Qiyao Wei, Samuel Holt, Jing Yang, Markus Wulfmeier, and Mihaela van der Schaar. 2025. The ai imperative: Scaling high-quality peer review in machine learning. find
  126. Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. 2025. Cycleresearcher: Improving automated research via automated review. find
  127. Po-Cheng Wu, An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2022. Incorporating peer reviews and rebuttal counter-arguments for meta-review generation. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM 2022), pages 2189–2198. Association for Computing… find
  128. Mike Thelwall and Abdullah Yaghi. 2024. Evaluating the predictive capacity of chatgpt for academic peer review outcomes across multiple platforms. find
  129. Sihong Wu, Yiling Ma, Yilun Zhao, Tiansheng Hu, Owen Jiang, Manasi Patwardhan, and Arman Cohan. 2026a. Rbtact: Rebuttal as supervision for actionable review feedback generation.
  130. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Jing Fu, Thomas Wolf, et al. 2023. Llama: Open and efficient foundation language models.
  131. Wenqing Wu, Yi Zhao, Yuzhuo Wang, Siyou Li, Juexi Shao, Yunfei Long, and Chengzhi Zhang. 2026b. Novbench: Evaluating large language models on academic paper novelty assessment.
  132. Keith Tyser, Ben Segev, Gaston Longhitano, Xin-Yu Zhang, Zachary Meeks, Jason Lee, Uday Garg, Nicholas Belsten, Avi Shporer, Madeleine Udell, Dov Te’eni, and Iddo Drori. 2024. Ai-driven review systems: Evaluating llms in scalable and bias-aware academic reviews. find
  133. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
  134. D. von Wedel, R. A. Schmitt, M. Thiele, R. Leuner, D. Shay, S. Redaelli, and M. S. Schaefer. 2024. Affiliation bias in peer review of abstracts by a large language model. JAMA, 331(3):252–253. find
  135. Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, Bo Zhang, Liqun Wei, Zhihao Sui, Wei Li, Botian Shi, Yu Qiao, Dahua Lin, and Conghui He. 2024. Mineru: An open-source solution for precise document content extraction. find
  136. Chengye Wang, Yifei Shen, Zexi Kuang, Arman Cohan, and Yilun Zhao. 2025. Sciver: Evaluating foundation models for multimodal scientific claim verification.
  137. Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. 2020. Reviewrobot: Explainable paper review generation based on knowledge synthesis. In Proceedings of the 13th International Conference on Natural Language Generation. Association for Computational Linguistics. find
  138. Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. 2025. Chain of draft: Thinking faster by writing less. find
  139. Rui Ye, Xianghe Pang, Jingyi Chai, Jiaao Chen, Zhenfei Yin, Zhen Xiang, Xiaowen Dong, Jing Shao, and Siheng Chen. 2024. Are we there yet? revealing the risks of utilizing large language models in scholarly peer review. find
  140. Jianxiang Yu, Zichen Ding, Jiaqi Tan, Kangyang Luo, Zhenmin Weng, Chenghua Gong, Long Zeng, Renjing Cui, Chengcheng Han, Qiushi Sun, Zhiyong Wu, Yunshi Lan, and Xiang Li. 2024. Automated peer reviewing in paper sea: Standardization, evaluation, and analysis. find
  141. Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2021. Can we automate scientific reviewing? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6269–6284. find
  142. Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can We Automate Scientific Reviewing? Journal of Artificial Intelligence Research, 75:171–212. Introduces the ASAP-Review dataset. find
  143. Qi Zeng, Meng Jiang, Shasha Li, and Zhou Yu. 2023. Meta-review generation with checklist-guided iterative introspection. arXiv preprint arXiv:2305.14647. — [arXiv:2305.14647](https://arxiv.org/abs/2305.14647) find
  144. Qi Zeng, Mankeerat Sidhu, Ansel Blume, Hou Pong Chan, Lu Wang, and Heng Ji. 2024. Scientific opinion summarization: Paper meta-review generation dataset, methods, and evaluation. find
  145. Sihang Zeng, Kai Tian, Kaiyan Zhang, Yuru wang, Junqi Gao, Runze Liu, Sa Yang, Jingxuan Li, Xinwei Long, Jiaheng Ma, Biqing Qi, and Bowen Zhou. 2025. Reviewrl: Towards automated scientific review with rl.
  146. Daoze Zhang, Zhijian Bao, Sihang Du, Zhiyi Zhao, Kuangling Zhang, Dezheng Bao, and Yang Yang. 2025a. Re 2 : A consistency-ensured dataset for full-stage peer review and multi-turn rebuttal discussions. find
  147. Jiayao Zhang, Hongming Zhang, Zhun Deng, and Dan Roth. 2022. Investigating fairness disparities in peer review: A language model enhanced approach. Introduces the ICLR-DB dataset (10k submissions, 36k reviews, 68k responses from ICLR 2017–2022). find
  148. Lili Zhang, Haomiaomiao Wang, Long Cheng, Libao Deng, and Tomas Ward. 2025b. Adversarial testing in llms: Insights into decision-making vulnerabilities. find
  149. Tianmai M. Zhang and Neil F. Abernethy. 2025. Reviewing scientific papers for critical problems with reasoning llms: Baseline approaches and automatic evaluation.
  150. Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. 2025b. Deepreview: Improving LLMbased paper review with humanlike deep thinking process. Introduces the DeepReview-13K dataset. find
  151. Zhenzhen Zhuang, Jiandong Chen, Hongfeng Xu, Yuwen Jiang, and Jialiang Lin. 2025. Large language models for automated scholarly paper review: A survey. Information Fusion, 124:103332. find
  152. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. Fine-tuning language models from human preferences. find
  153. Zhuoyang Zou, Abolfazl Ansari, Delvin Ce Zhang, Dongwon Lee, and Wenpeng Yin. 2026. Diagpaper: Diagnosing valid and specific weaknesses in scientific papers via multi-agent reasoning. find
  154. Ádám Kovács and Gábor Recski. 2025. Lettucedetect: A hallucination detection framework for rag applications. find
  155. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. find
  156. Wayne Xin Zhao, Kun Zhou, Junlin Yao, Zhicheng Dou, and Ji-Rong Wen. 2023. A survey of large language models.
  157. Yilun Zhao, Kaiyan Zhang, Tiansheng Hu, Sihong Wu, Ronan Le Bras, Taira Anderson, Jonathan Bragg, Joseph Chee Chang, Jesse Dodge, Matt Latzke, Yixin Liu, Charles McGrady, Xiangru Tang, Zihang Wang, Chen Zhao, Hannaneh Hajishirzi, Doug Downey, and Arman Cohan. 2025. Sciarena: An open evaluation platf… find
  158. Ruiyang Zhou, Lu Chen, and Kai Yu. 2024a. Is LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9340– 9351,…
  159. Ruiyang Zhou, Lu Chen, and Kai Yu. 2024b. Is LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). Introduces the RR-…
  160. Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. 2025a. Deepreview: Improving llm-based paper review with human-like deep thinking process. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-can-ai-be-good-peer/