Elamur Research⌕
☰
home / papers / 2026 / Can AI Revise Research Papers with Human Review Feedback? An Empirical Study and Benchmark

Can AI Revise Research Papers with Human Review Feedback? An Empirical Study and Benchmark

Paper · 2026 · Zihan Luo, Hong Huang, Jianxun Lian et al.

Summary

The paper asks whether LLM-based agents can take a submitted manuscript plus reviewer criticism and produce an improved version, as authors do before camera-ready. To test this, the authors build ReviseBench from 12 papers accepted at ICLR 2025. Each instance pairs the reviewers' listed weaknesses from OpenReview with a GitHub code snapshot and the LaTeX sources, both taken from before reviews were released, which prevents leakage. Agents running in OpenHands must set up environments, run new experiments, and rewrite the text. Their output is compared pairwise against the authors' camera-ready version by an LLM judge, which was validated on a JudgeTest set. A companion platform, ReviseArena, ranks models against each other. Results are poor. The best system beats the human revision in fewer than 10% of comparisons. Models tend to make small, shallow edits, write unprofessionally, break layouts, and sometimes invent experimental numbers. The benchmark shows that AI-assisted revision still needs close human supervision.

Key points

  • ReviseBench has 12 ICLR 2025 papers, each with three components: review weaknesses (review.md), a pre-review GitHub commit, and the initial LaTeX source. It is designed to grow through agent-driven automatic data collection.
  • Evaluation is staged. It starts with basic executability, such as whether a PDF compiles, and moves up to quality judged by an LLM through pairwise comparison with the camera-ready version. ReviseArena adds model-versus-model battles.
  • Against the human revisions, GPT-5 wins 8.33% of comparisons and DeepSeek-V3.2 wins 0%. Other models tested include o3, o4-mini, Gemini-2.5-Pro and Claude-4-Sonnet.
  • Common failures include:
    • fewer references, tables and figures than the human revisions;
    • rebuttal-style remarks left in the body text;
    • tables overflowing the margins and text overlapping figures.
  • Fabrication was checked in two steps, with the second step using Gemini-3-Pro to audit agent trajectories. Claude-4-Sonnet fabricated data in 5 of 6 cases where it added new results, and DeepSeek-V3.2 in 1 of 2. Any fabricated paper is counted as an invalid PDF.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Microsoft Research AsiaBeijing · China73 works · 60,539 citations🎓Huazhong University of Science and TechnologyWuhan · China45 works · 6,345 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksHistory and Philosophy of Science · 53 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Zihan Luo, Hong Huang, Jianxun Lian, Yu Chang, Xing Xie and Hai Jin. 2026. Can AI Revise Research Papers with Human Review Feedback? An Empirical Study and Benchmark. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.887

@inproceedings{luo2026can,
  title = {{Can AI Revise Research Papers with Human Review Feedback? An Empirical Study and Benchmark}},
  author = {Zihan Luo, Hong Huang, Jianxun Lian, Yu Chang, Xing Xie and Hai Jin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.887},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.887}
}

References

33 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. 2025. Researchagent: Iterative research idea generation over scientific literature with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lingu…
  2. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. 2025. Mle-bench: Evaluating machine learning agents on machine learning engineering. In Proceedings of the 13th Internation…
  3. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. 2025. Scienceagentbench: Toward rigorous assessme… find
  4. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In Proceedings of the 41st International Co…
  5. Huawen Feng, Pu Zhao, Qingfeng Sun, Can Xu, Fangkai Yang, Lu Wang, Qianli Ma, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2025. Warriorcoder: Learning from expert battles to augment code large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computati… find
  6. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2024. Mlagentbench: Evaluating language agents on machine learning experimentation. In Proceedings of the 41st International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.
  7. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Livecodebench: Holistic and contamination free evaluation of large language models for code. In Proceedings of the 13th International Conference on Learning Repr…
  8. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In Proceedings of the 12th International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,…
  9. Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang. 2024. Agentreview: Exploring peer review dynamics with LLM agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pa… find
  10. Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. 2025. Dsbench: How far are data science agents from becoming data science experts? In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, Singapore… find
  11. Patrick Tser Jern Kon, Qiuyi Ding, Jiachen Liu, Xinyi Zhu, Jingjia Peng, Jiarong Xing, Yibo Huang, Yiming Qiu, Jayanth Srinivasa, Myungjin Lee, Mosharaf Chowdhury, Matei Zaharia, and Ang Chen. 2026. EXP-bench: Can AI conduct AI research experiments? In Proceedings of the 14th International Conferenc… find
  12. Byung Cheol Lee and Jaeyeon Chung. 2024. An empirical investigation of the impact of chatgpt on creativity. Nature Human Behaviour, 8(10):1906–1914. find
  13. Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. 2024. Split and merge: Aligning position biases in llm-based evaluators. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16,… find
  14. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025. Deepseek-v3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
  15. Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune. 2026. Towards end-to-end automation of ai research. Nature, 651:914 – 919. find
  16. Zhi-Cun Lyu, Xinye Li, Zheng Xie, and Ming Li. 2025. Top pass: Improve code generation by pass@k-maximized code ranking. Frontiers of Computer Science, 19(8):198341. find
  17. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. 2025. Agent laboratory: Using LLM agents as research assistants. In Findings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Su…
  18. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2025. Can llms generate novel research ideas? A large-scale human study with 100+ NLP researchers. In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.
  19. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. 2025. Paperbench: Evaluating AI’s ability to replicate AI research. In Proceedings of the 42nd Inte… find
  20. Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. 2025a. AI-researcher: Autonomous scientific innovation. In Proceedings of the 39th Annual Conference on Neural Information Processing Systems, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025. find
  21. Xiangru Tang, Yuliang Liu, Zefan Cai, Daniel Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, and 1 others. 2025b. Ml-bench: Evaluating large language models and agents for machine learning tasks on repository-level code. In ICLR 2025 Third Workshop on Deep Learning for…
  22. Haoran Wang, Zhenyu Hou, Yao Wei, Jie Tang, and Yuxiao Dong. 2025a. SWE-Dev: Building software engineering agents with training and inference scaling. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27-August 1, 2025, pages 3742–3761. Association for Com… find
  23. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, and 2 others. 2025b. Openhands: An open platform…
  24. Ryan Watkins. 2024. Guidance for researchers and peer-reviewers on the ethical use of large language models (LLMs) in scientific research workflows. AI and Ethics, 4(4):969–974. find
  25. Shihao Weng, Yang Feng, Yining Yin, Zhenlun Zhang, and Baowen Xu. 2026. Data preparation and quality for code-centric generative software engineering tasks: a systematic literature review. Frontiers of Computer Science, 20(9):2009203. find
  26. Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. 2025. Cycleresearcher: Improving automated research via automated review. In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenRevi… find
  27. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. 2024. Large language models for automated open-domain scientific hypotheses discovery. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, p… find
  28. Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, and Dongzhan Zhou. 2025. Moose-chem: Large language models for rediscovering unseen chemistry scientific hypotheses. In Proceedings of the 13th International Conference on Learning Representations, I… find
  29. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, and 1 others. 2025. GLM-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471.
  30. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th Annual Conference on Neural Inf…
  31. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2025a. Judgelm: Fine-tuned large language models are scalable judges. In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.
  32. Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. 2025b. Deepreview: Improving llm-based paper review with human-like deep thinking process. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27-August 1, 2025, pages 29330– 2… find
  33. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, and 2 others. 2025. Bigcodebench:…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2026-can-ai-revise-research-papers/