Elamur Research⌕
☰
home / papers / 2025 / TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review

TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review

Paper · 2025 · Yuan Chang, Ziyue Li, Hengyuan Zhang et al.

Summary

Peer review is under strain from growing submission volumes. LLM-based review assistants tend to miss details buried in long papers, give shallow critiques, or, as with multi-agent systems like MARG, consume very large numbers of tokens. TreeReview addresses this by treating reviewing as question answering organised in a tree. In a top-down pass, a question-generator agent uses the paper's title, abstract and table of contents to split the root task ("write a review") into progressively narrower sub-questions. In a bottom-up pass, an answer-synthesizer agent answers the leaves from relevant paper chunks and merges the answers upward into a final review. Along the way it can add follow-up questions where deeper probing seems warranted. The authors also release a benchmark built from ICLR and NeurIPS papers and their human reviews, covering two tasks: full review generation and actionable feedback comment generation. Both LLM-judge and expert evaluations favour TreeReview over strong baselines, and it needs far less computation. This makes deep automated review feedback more affordable. The authors present it as a tool for authors and reviewers, not a replacement for human judgment.

Key points

  • On full review generation, LLM-as-Judge scores beat the best baseline by 12.27% on specificity, 11.22% on comprehensiveness and 6.45% on technical depth.
  • On feedback comment generation, TreeReview has the highest precision and exceeds MARG by 5.7% in Jaccard.
  • Average usage is about 459K tokens per paper, compared with about 2.31M for MARG (80.2% less) and about 1.01M for MARG-BASE (54.4% less).
  • In ablations, removing the tree (TreeReview−DEC) hurts results substantially. Removing dynamic expansion (TreeReview−EXP) lowers Jaccard from 15.33% to 12.98% and ITF-IDF from 4.62 to 4.01.
  • Expansion is triggered for about 38.5% of non-leaf questions, adding roughly 25.6 questions per review. Code and benchmark are available at github.com/YuanChang98/tree-review.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🎓University of Hong KongPok Fu Lam · China89 works · 11,774 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🎓The Hong Kong University of Science and Technology (Guangzhou)Guangzhou · China53 works · 1,367 citations

Also on the byline: National Science Library.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksHistory and Philosophy of Science · 53 works

Cite

Cite as Yuan Chang, Ziyue Li, Hengyuan Zhang, Yuanbo Kong, Yanru Wu, Hayden Kwok-Hay So, Zhijiang Guo, Liya Zhu and Ngai Wong. 2025. TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.790

@inproceedings{chang2025treereview,
  title = {{TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review}},
  author = {Yuan Chang, Ziyue Li, Hengyuan Zhang, Yuanbo Kong, Yanru Wu, Hayden Kwok-Hay So, Zhijiang Guo, Liya Zhu and Ngai Wong},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.790},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.790}
}

References

66 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eric Chamoun, Michael Schlichtkrull, and Andreas Vlachos. 2024. Automated focused feedback generation for scientific writing assistance. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9742–9763, Bangkok, Thailand. Association for Computational Linguistics. find
  2. Yuan Chang, Ziyue Li, and Xiaoqiu Le. 2024. Guiding large language models via external attention prompting for scientific extreme summarization. In Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024), pages 226–242, Bangkok, Thailand. Association for Computational Linguist… find
  3. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. Marg: Multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04259. find
  4. Zhongfen Deng, Hao Peng, Congying Xia, Jianxin Li, Lifang He, and Philip Yu. 2020. Hierarchical bidirectional self-attention networks for paper review rating recommendation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6302–6314, Barcelona, Spain (Online).… find
  5. Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, and 21 others. 2024. LLMs assist NLP resea…
  6. Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022. Successive prompting for decomposing complex questions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1251–1265, Abu Dhabi, United Arab Emirates. Association for Computational Lingui… find
  7. Tomáš Foltýnek, Norman Meuschke, and Bela Gipp. 2019. Academic plagiarism detection: A systematic literature review. ACM Comput. Surv., 52(6). find
  8. Zhaolin Gao, Kianté Brantley, and Thorsten Joachims. 2024. Reviewer2: Optimizing review generation through prompt generation. arXiv preprint arXiv:2402.10886.
  9. Seth S Leopold. 2015. Increased manuscript submissions prompt journals to make hard choices. Clinical Orthopaedics and Related Research®, 473(3):753– 755. find
  10. Robert E Gropp, Scott Glisson, Stephen Gallo, and Lisa Thompson. 2017. Peer review: A system under stress. BioScience, 67(5):407–410. find
  11. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, and 1 others. 2024a. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594. find
  12. Xiao-Yu Guo, Yuan-Fang Li, and Gholamreza Haffari. 2022. Complex reading comprehension through question decomposition. In Proceedings of the 20th Annual Workshop of the Australasian Language Technology Association, pages 31–40, Adelaide, Australia. Australasian Language Technology Association. find
  13. Daniil Gurgurov, Tanja Bäumel, and Tatiana Anikina. 2024. Multilingual large language models and curse of multilinguality. arXiv preprint arXiv:2406.10602. — [arXiv:2406.10602](https://arxiv.org/abs/2406.10602) find
  14. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023.
  15. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:…
  16. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Comp…
  17. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022. Decomposed prompting: A modular approach for solving complex tasks. CoRR, abs/2210.02406. find
  18. Ari Kobren, Barna Saha, and Andrew McCallum. 2019. Paper matching with local fairness constraints. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, page 1247–1257, New York, NY, USA. Association for Computing Machinery. find
  19. Peder Larsen and Markus Von Ins. 2010. The rate of growth in scientific publication and the decline in coverage provided by science citation index. Scientometrics, 84(3):575–603. find
  20. Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2024b. LooGLE: Can long-context language models understand long contexts? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16304–16333, Bangkok, Thailand. Association for…
  21. Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2024c. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060.
  22. Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong, Zhijiang Guo, Le Song, and Cheng-Lin Liu. 2025a. From system 1 to system 2: A survey of reasoning large language models. CoRR…
  23. Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, and 1 others. 2025b. From system 1 to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502.17419.
  24. Ziyue Li, Yuan Chang, and Xiaoqiu Le. 2024d. Simulating expert discussions with multi-agent for enhanced scientific problem solving. In Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024), pages 243–256, Bangkok, Thailand. Association for Computational Linguistics. find
  25. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, and 1 others. 2024a. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8):AIoa2400196. find
  26. Xiao Liang, Xinyu Hu, Simiao Zuo, Yeyun Gong, Qiang Lou, Yi Liu, Shao-Lun Huang, and Jian Jiao. 2024b. Task oriented in-domain data augmentation. arXiv preprint arXiv:2406.16694. find
  27. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  28. Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. 2023a. Automated scholarly paper review: Concepts, technologies, and challenges. Information fusion, 98:101830. find
  29. Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. 2023b. Moprd: A multidisciplinary open peer review dataset. Neural Computing and Applications, 35(34):24191–24206. find
  30. Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, and 1 others. 2025a. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407.
  31. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  32. Weihao Liu, Ning Wu, Shiping Yang, Wenbiao Ding, Shining Liang, Ming Gong, and Dongmei Zhang. 2025b. Mudaf: Long-context multi-document attention focusing through contrastive learning on attention heads. arXiv preprint arXiv:2502.13963. — [arXiv:2502.13963](https://arxiv.org/abs/2502.13963) find
  33. Renze Lou, Hanzi Xu, Sijia Wang, Jiangshu Du, Ryo Kamoi, Xiaoxin Lu, Jian Xie, Yuxuan Sun, Yusen Zhang, Jihyun Janice Ahn, Hongchao Fang, Zhuoyang Zou, Wenchao Ma, Xi Li, Kai Zhang, Congying Xia, Lifu Huang, and Wenpeng Yin. 2024. AAAR-1.0: Assessing AI’s potential to assist research. In 2nd AI4Rese… find
  34. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292.
  35. Qusay H Mahmoud and 1 others. 2024. Evaluating the efficacy of large language models in automating academic peer reviews. In 2024 International Conference on Machine Learning and Applications (ICMLA), pages 1208–1213. IEEE. find
  36. Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8864–8880, Online. Association for Computational Linguis…
  37. Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. 2024. ADaPT: As-needed decomposition and planning with language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4226–4252, Mexico City, Mexico. As…
  38. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, Singapore. Association for Computational Linguistics. find
  39. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  40. Patrick E Shrout and Joseph L Fleiss. 1979. Intraclass correlations: uses in assessing rater reliability. Psychological bulletin, 86(2):420. find
  41. Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, and Linqi Song. 2024. Proxyqa: An alternative framework for evaluating long-form text generation with large language models. In Proceedings of the 62nd Annual Meeting of the Associ…
  42. Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. 2024. Can llms learn from previous mistakes? investigating llms’ errors to boost for reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3065–30…
  43. Guanchao Wang, Pawin Taechoyotin, Tong Zeng, Bradley Sides, and Daniel Acuna. 2024a. MAMORX: Multi-agent multi-modal scientific review generation with external knowledge. In Neurips 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenges. find
  44. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… find
  45. Sizhe Wang, Yongqi Tong, Hengyuan Zhang, Dawei Li, Xin Zhang, and Tianlong Chen. 2024b. Bpo: Towards balanced preference optimization between knowledge breadth and depth in alignment. arXiv preprint arXiv:2411.10914. find
  46. Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Qingsong Wen, Wei Ye, and 1 others. 2024c. Autosurvey: Large language models can automatically write surveys. Advances in Neural Information Processing Systems, 37:115119–115145. find
  47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  48. Kun Zhang, Jiali Zeng, Fandong Meng, Yuanzhuo Wang, Shiqi Sun, Long Bai, Huawei Shen, and Jie Zhou. 2024c. Tree-of-reasoning question decomposition for complex question answering with large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19560–19568. find
  49. Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. 2025. Cycleresearcher: Improving automated research via automated review. In The Thirteenth International Conference on Learning Representations. find
  50. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  51. Hang Yang, Hao Chen, Hui Guo, Yineng Chen, Ching-Sheng Lin, Shu Hu, Jinrong Hu, Xi Wu, and Xin Wang. 2024a. Llm-medqa: Enhancing medical question answering through case studies in large language models. arXiv preprint arXiv:2501.05464. — [arXiv:2501.05464](https://arxiv.org/abs/2501.05464) find
  52. Ruihan Yang, Jiangjie Chen, Yikai Zhang, Siyu Yuan, Aili Chen, Kyle Richardson, Yanghua Xiao, and Deqing Yang. 2024b. Selfgoal: Your language agents already know how to achieve high-level goals. In NeurIPS 2024 Workshop on Open-World Agents. find
  53. Shiping Yang, Jie Wu, Wenbiao Ding, Ning Wu, Shining Liang, Ming Gong, Hengyuan Zhang, and Dongmei Zhang. 2024c. Quantifying the robustness of retrieval-augmented language models against spurious features in grounding data. arXiv preprint arXiv:2503.05587. — [arXiv:2503.05587](https://arxiv.org/abs/2503.05587) find
  54. Jianxiang Yu, Zichen Ding, Jiaqi Tan, Kangyang Luo, Zhenmin Weng, Chenghua Gong, Long Zeng, Ren-Jing Cui, Chengcheng Han, Qiushi Sun, Zhiyong Wu, Yunshi Lan, and Xiang Li. 2024. Automated peer reviewing in paper SEA: Standardization, evaluation, and analysis. In Findings of the Association for Compu… find
  55. Yiyao Yu, Yuxiang Zhang, Dongdong Zhang, Xiao Liang, Hengyuan Zhang, Xingxing Zhang, Ziyi Yang, Mahmoud Khademi, Hany Awadalla, Junjie Wang, and 1 others. 2025. Chain-of-reasoning: Towards unified mathematical reasoning in large language models via a multi-paradigm perspective. arXiv preprint arXiv:…
  56. Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can we automate scientific reviewing? J. Artif. Int. Res., 75. find
  57. Hengyuan Zhang, Chenming Shang, Sizhe Wang, Dongdong Zhang, Feng Yao, Renliang Sun, Yiyao Yu, Yujiu Yang, and Furu Wei. 2024a. Shifcon: Enhancing non-dominant language capabilities with a shift-based contrastive framework. arXiv preprint arXiv:2410.19453. — [arXiv:2410.19453](https://arxiv.org/abs/2410.19453) find
  58. Hengyuan Zhang, Yanru Wu, Dawei Li, Sak Yang, Rui Zhao, Yong Jiang, and Fei Tan. 2024b. Balancing speciality and versatility: a coarse to fine framework for supervised fine-tuning large language model. In Findings of the Association for Computational Linguistics ACL 2024, pages 7467–7509. find
  59. Qi Zhao, Haotian Fu, Chen Sun, and George Konidaris. 2024. EPO: Hierarchical LLM agents with environment preference optimization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6401–6415, Miami, Florida, USA. Association for Computational Linguistics… find
  60. Wenqing Zheng, SP Sharan, Ajay Kumar Jaiswal, Kevin Wang, Yihan Xi, Dejia Xu, and Zhangyang Wang. 2023a. Outline, then details: Syntactically guided coarse-to-fine code generation. In International Conference on Machine Learning, pages 42403–42419. PMLR. find
  61. Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh TN Nguyen, Lauren T May, Geoffrey I Webb, and Shirui Pan. 2023b. Large language models for scientific synthesis, inference and explanation. arXiv preprint arXiv:2310.07984. — [arXiv:2310.07984](https://arxiv.org/abs/2310.07984) find
  62. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representation…
  63. Haichen Zhou, Xiaorong Huang, Hongjun Pu, and Zhang Qi. 2024a. May generative ai be a reviewer on an academic paper? find
  64. Ruiyang Zhou, Lu Chen, and Kai Yu. 2024b. Is LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9340– 9351,…
  65. Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, and 1 others. 2024c. A survey on efficient inference for large language models. arXiv preprint arXiv:2404.14294. find
  66. Zhenzhen Zhuang, Jiandong Chen, Hongfeng Xu, Yuwen Jiang, and Jialiang Lin. 2025. Large language models for automated scholarly paper review: A survey. Information Fusion, 124:103332. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chang-2025-treereview-dynamic-tree-questions-framework/