Elamur Research⌕
☰
home / papers / 2025 / REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning

REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning

Paper · 2025 · Pawin Taechoyotin and Daniel Acuna

Summary

LLM-based peer review tools tend to give vague, flattering feedback. They are also hard to judge without a human reference review. This paper asks whether an explicitly reasoning model, tuned with reinforcement learning toward several quality criteria, can do better. The authors built PeerRT, which combines ICLR 2017–2020 reviews, full paper texts extracted with GROBID, and synthetic thinking traces produced by Claude Sonnet 3.7. They also designed HPRR, a reward that scores review sentences on aspects such as criticism, examples, importance and suggestions, and adds a METEOR-based relevance term linking the review to the manuscript. DeepSeek-R1-Distill-Qwen-7B was first adapted with LoRA SFT and then optimized with GRPO under two weightings, human-aligned (REMOR-H) and uniform (REMOR-U). Both variants reached roughly double the average reward of human reviews, agentic systems and commercial LLMs, and they avoided weak outliers. Each review took about a minute to generate. The authors release the reward, the data and the models. A key caveat is that all evaluation relies on the constructed reward rather than expert judgment.

Key points

  • PeerRT contains 16.8K ICLR reviews of 5.5K submissions, enriched with OpenReview full texts and Claude Sonnet 3.7 reasoning traces.
  • The training pipeline is LoRA SFT followed by GRPO on DeepSeek-R1-Distill-Qwen-7B, which yields REMOR-H and REMOR-U.
  • The human-aligned weights penalized criticism and gave too much weight to the METEOR relevance term. As a result, REMOR-H reviews were long and repetitive, and REMOR-U was judged qualitatively better.
  • REMOR-U had the highest mean and the lowest variance of normalized rewards, ahead of MARG-S, MAMORX and general LLM baselines. The SFT-only model came second.
  • Limitations include using only text input, a restricted context window, a small pool of preference votes behind HPRR, and the absence of a blinded expert evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 worksHistory and Philosophy of Science · 83 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Pawin Taechoyotin and Daniel Acuna. 2025. REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning. arXiv:2505.11718.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.11718 · arXiv 2505.11718

@misc{taechoyotin2025remor,
  title = {{REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning}},
  author = {Pawin Taechoyotin and Daniel Acuna},
  year = {2025},
  doi = {10.48550/arxiv.2505.11718},
  eprint = {2505.11718},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.11718}
}

References

54 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005.
  2. Alberto Bartoli, Andrea De Lorenzo, Eric Medvet, and Fabiano Tarlao. Your paper has been accepted, rejected, or whatever: Automatic generation of scientific paper reviews. In Availability, Reliability, and Security in Information Systems: IFIP WG 8.4, 8.9, TC 5 International Cross-Domain Conference,… find
  3. Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. ISSN 00063444, 14643510. URL http://www.jstor.org/stable/2334029. ↗
  4. Marijke Breuning, Jeremy Backstrom, Jeremy Brannon, Benjamin Isaak Gross, and Michael Widmeier. Reviewer fatigue? why scholars decline to review their peers’ work. PS: Political Science & Politics, 48(4):595–600, 2015. find
  5. Andrea Castelletti, Francesca Pianosi, and Marcello Restelli. A multiobjective reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single run. Water Resources Research, 49(6):3476–3486, 2013. find
  6. A Checco, L Bracciale, P Loreti, S Pinfield, and G Bianchi. Ai-assisted peer review. humanities and social sciences communications, 8 (1), 1-11, 2021. find
  7. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning, 2…
  8. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. find
  9. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. Marg: Multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04259, 2024.
  10. Rafael D’Andrea and James P O’Dwyer. Can editors save peer review from peer reviewers? PloS one, 12(10):e0186111, 2017. find
  11. Nady El-Guebaly, John Foster, Anees Bahji, and Matilda Hellman. The critical role of peer reviewers: Challenges and future steps. Nordic Studies on Alcohol and Drugs, 40(1):14–21, 2023. find
  12. Stephen Gallo, Lisa Thompson, Karen Schmaling, and Scott Glisson. Risk evaluation in peer review of grant applications. Environment Systems and Decisions, 38:216–229, 2018. find
  13. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  14. Cigdem Kadaifci, Erkan Isikli, and Y Ilker Topcu. Fundamental problems in the peer-review process and stakeholders’ perceptions of potential suggestions for improvement. Learned Publishing, 38(1):e1637, 2025. find
  15. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine Van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. A dataset of peer reviews (peerread): Collection, insights and nlp applications. arXiv preprint arXiv:1804.09635, 2018.
  16. Ingvar Karlberg. On peer review–the cornerstone of scientific publication, 2015. find
  17. Jacalyn Kelly, Tara Sadeghieh, and Khosrow Adeli. Peer review in scientific publications: benefits, critiques, & a survival guide. Ejifcc, 25(3):227, 2014. find
  18. Chhavi Kirtani, Madhav Krishan Garg, Tejash Prasad, Tanmay Singhal, Murari Mandal, and Dhruv Kumar. Revieweval: An evaluation framework for ai-generated reviews. arXiv preprint arXiv:2502.11736, 2025. find
  19. Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective optimization. IEEE transactions on cybernetics, 51(6):3103–3114, 2020. find
  20. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8): AIoa2400196, 2024.
  21. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.
  22. Ryan Liu and Nihar B Shah. Reviewergpt? an exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306.00622, 2023.
  23. Patrice Lopez. Grobid: Combining automatic bibliographic data recognition and term extraction for scholarship publications. In International conference on theory and practice of digital libraries, pages 473–474. Springer, 2009. find
  24. Christopher P Morley and Sam Grammer. Now more than ever: reflections on the state and importance of peer review. PRiMER: Peer-review reports in medical education research, 5:36, 2021. find
  25. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.
  26. Syavash Nobarany and Kellogg S Booth. Use of politeness strategies in signed open peer review. Journal of the Association for Information Science and Technology, 66(5):1048–1064, 2015. find
  27. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training… — Training language models to follow instructions with human feedback
  28. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. find
  29. David B Resnik, Christina Gutierrez-Ford, and Shyamal Peddada. Perceptions of ethical problems with scientific journal peer review: an exploratory study. Science and engineering ethics, 14:305–310, 2008. find
  30. Laurent Romary and Patrice Lopez. Grobid-information extraction from scientific publications. ERCIM News, 100, 2015. find
  31. Abu-Ubaida Sani. Journal peer review: Meaning, types, and processes. Journal Peer Review: Meaning, 2023. find
  32. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. find
  33. Anna Severin, Michaela Strinzel, Matthias Egger, Tiago Barros, Alexander Sokolov, Julia Vil-strup Mouatt, and Stefan Müller. Relationship between journal impact factor and the thoroughness and helpfulness of peer reviews. PLOS Biology, 21(8):1–18, 08 2023. doi: 10.1371/journal. pbio.3002238. URL htt… — [doi:10.1371/journal](https://doi.org/10.1371/journal) find
  34. Nihar B Shah. Challenges, experiments, and computational solutions in peer review. Communications of the ACM, 65(6):76–87, 2022. find
  35. Minghao Shao, Abdul Basit, Ramesh Karri, and Muhammad Shafique. Survey of different large language model architectures: Trends, benchmarks, and challenges. IEEE Access, 2024. find
  36. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
  37. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017. find
  38. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://arxiv.org/abs/2408.03314. ↗
  39. Cecilia Superchi, José Antonio González, Ivan Solà, Erik Cobo, Darko Hren, and Isabelle Boutron. Tools used to assess the quality of peer review reports: a methodological systematic review. BMC Medical Research Methodology, 19(1):48, Mar 2019. ISSN 1471-2288. doi: 10.1186/s12874-019-0688-x. URL http… — [doi:10.1186/s12874-019-0688-x](https://doi.org/10.1186/s12874-019-0688-x) find
  40. Cecilia Superchi, Darko Hren, David Blanco, Roser Rius, Alessandro Recchioni, Isabelle Boutron, and José Antonio González. Development of arcadia: a tool for assessing the quality of peer-review reports in biomedical research. BMJ Open, 10(6), 2020. ISSN 2044-6055. doi: 10.1136/bmjopen-2019-035604.… — [doi:10.1136/bmjopen-2019-035604](https://doi.org/10.1136/bmjopen-2019-035604) find
  41. Pawin Taechoyotin, Guanchao Wang, Tong Zeng, Bradley Sides, and Daniel Acuna. Mamorx: Multi-agent multi-modal scientific review generation with external knowledge. In Neurips 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenges, 2024. find
  42. Misha Teplitskiy, Hao Peng, Andrea Blasco, and Karim R Lakhani. Is novel research worth doing? evidence from peer review at 49 journals. Proceedings of the National Academy of Sciences, 119(47):e2118046119, 2022. find
  43. Gang Wang, Qi Peng, Yanfeng Zhang, and Mingyang Zhang. What have we learned from openreview? World Wide Web, 26(2):683–708, Mar 2023. ISSN 1573-1413. doi: 10.1007/ s11280-022-01109-z. URL https://doi.org/10.1007/s11280-022-01109-z. — [doi:10.1007/s11280-022-01109-z](https://doi.org/10.1007/s11280-022-01109-z) ↗
  44. Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. Reviewrobot: Explainable paper review generation based on knowledge synthesis. arXiv preprint arXiv:2010.06119, 2020. find
  45. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Proce…
  46. Christie Wilcox. Rude reviews are pervasive and sometimes harmful, study finds. Science, 366 (6472):1433–1433, 2019. find
  47. Weizhe Yuan, Pengfei Liu, and Graham Neubig. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171–212, 2022.
  48. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. find
  49. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39–57, 2024.
  50. Youming Zhang. A novel multi-objective deep q-network: Addressing immediate and delayed rewards in multi-objective q-learning. IEEE Access, 12:144932–144949, 2024. find
  51. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
  52. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangko…
  53. Ruiyang Zhou, Lu Chen, and Kai Yu. Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9340–9351, 2024.
  54. Zhenzhen Zhuang, Jiandong Chen, Hongfeng Xu, Yuwen Jiang, and Jialiang Lin. Large language models for automated scholarly paper review: A survey. arXiv preprint arXiv:2501.10326, 2025. — [arXiv:2501.10326](https://arxiv.org/abs/2501.10326) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/taechoyotin-2025-remor-automated-peer-review-generation/