Elamur Research⌕
☰
home / papers / 2025 / Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition

Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition

Paper · 2025 · Xuemei Tang, Xufeng Duan and Zhenguang Cai

Summary

Researchers increasingly ask plain chatbots without retrieval support to help draft literature reviews. Yet nobody has systematically measured how trustworthy these outputs are. The authors build a fully automatic benchmark with no human in the loop. They use 1,105 expert-written reviews published in 2023 across 51 Annual Reviews journals, covering six disciplines, as the gold standard.

Each model is tested on three separate tasks:

  • listing relevant references with full bibliographic metadata, given a title and keywords;
  • drafting an abstract from the title and keywords;
  • composing a roughly 1000-word cited review from the title, keywords and abstract.

Citations are checked against Semantic Scholar to estimate hallucination. The generated prose is compared with the human originals for semantic coverage and, using NLI models, for factual consistency. Five LLMs were tested, and every one of them still invents references. Quality also shifts noticeably from one field to another.

The work matters because it offers a reusable, scalable way to quantify how reliable LLMs are for academic writing.

Key points

  • The dataset is 1,105 Annual Reviews articles from 2023, covering 51 journals and six disciplines. Data and code are released on GitHub (tangxuemei1995/Eval_LLM_Literature_Review).
  • The evaluated models are Claude 3.5 Sonnet, DeepSeek-V3, GPT-4o, Qwen-2.5-72B and LLaMA-3.2-3B. An extra contamination check tested GPT-5 on 2025 articles.
  • Reference metrics include precision, overlap with human-cited works, and a title search rate. Review composition is additionally scored with KPR.
  • Mathematics gave the highest reference precision for most models, while GPT-4o did best in Social Science. Precision correlates positively with how often the correct references are cited (0.4 for Claude-3.5, 0.51 for DeepSeek-V3).
  • The automatic hallucination check was validated against majority-vote judgments from three annotators on 100 references. It reached kappa 0.71 and 86% accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chinese University of Hong KongHong Kong · China126 works · 14,073 citations🎓Hong Kong Polytechnic UniversityHong Kong · China84 works · 4,956 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 worksHistory and Philosophy of Science · 53 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Xuemei Tang, Xufeng Duan and Zhenguang Cai. 2025. Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.83

@inproceedings{tang2025large,
  title = {{Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition}},
  author = {Xuemei Tang, Xufeng Duan and Zhenguang Cai},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.83},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.83}
}

References

18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shubham Agarwal, Gaurav Sahu, Abhay Puri, Issam H. Laradji, Krishnamurthy DJ Dvijotham, Jason Stanley, Laurent Charlin, and Christopher Pal. 2024. Llms for literature review: Are we there yet? (arXiv:2412.15249). ArXiv:2412.15249 [cs].
  2. Ayush Agrawal, Mirac Suzgun, Lester Mackey, and Adam Tauman Kalai. 2024. Do language models know when they’re hallucinating references? find
  3. Muhammad Bello Aliyu, Rahat Iqbal, and Anne James. 2018. The canonical model of structure for data extraction in systematic reviews of scientific research articles. In 2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS), page 264–271. find
  4. Fadi Aljamaan, Mohamad-Hani Temsah, Ibraheem Altamimi, Ayman Al-Eyadhy, Amr Jamal, Khalid Alhasan, Tamer A. Mesallam, Mohamed Farahat, and Khalid H. Malki. 2024. Reference hallucination score for medical artificial intelligence chatbots: Development and usability study. JMIR Medical Informatics, 12(… find
  5. Sai Anirudh Athaluri, Sandeep Varma Manthena, V S R Krishna Manoj Kesapragada, Vineel Yarlagadda, Tirth Dave, and Rama Tulasi Siri Duddumpudi. 2023. Exploring the boundaries of reality: Investigating the phenomenon of artificial intelligence hallucination in scientific writing through chatgpt refere… find
  6. Mikaël Chelli, Jules Descamps, Vincent Lavoué, Christophe Trojani, Michel Azar, Marcel Deckert, Jean-Luc Raynier, Gilles Clowez, Pascal Boileau, and Caroline Ruetsch-Chelli. 2024. Hallucination rates and reference accuracy of chatgpt and bard for systematic reviews: Comparative analysis. Journal of… find
  7. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or llms as the judge? a study on judgement biases. (arXiv:2402.10669). ArXiv:2402.10669 [cs].
  8. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. (arXiv:2305.14627). ArXiv:2305.14627 [cs].
  9. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Associa…
  10. Linguistics: EMNLP 2024, page 4852–4872, Miami, Florida, USA. Association for Computational Linguistics. find
  11. Xing Shangyu, Zhao Fei, Wu Zhen, An Tuo, Chen Weihao, Li Chunhui, Zhang Jianbing, and Dai Xinyu. 2024. Efuf: Efficient fine-grained unlearning framework for mitigating hallucinations in multimodal large language models. page 1167–1181. find
  12. Jiyao Wang, Haolong Hu, Zuyuan Wang, Song Yan, Youyu Sheng, and Dengbo He. 2024a. Evaluating large language models on academic literature understanding and review: An empirical study among early-stage scholars. In Proceedings of the CHI Conference on Human Factors in Computing Systems, page 1–18, Ho… find
  13. Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang. 2024b. Autosurvey: Large language models can automatically write surveys. (arXiv:2406.10252). ArXiv:2406.10252. find
  14. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. (arXiv:2306.05685). ArXiv:2306.05685 [cs].
  15. Chao-Chun Hsu, Erin Bransom, Jenna Sparks, Bailey Kuehl, Chenhao Tan, David Wadden, Lucy Wang, and Aakanksha Naik. 2024. CHIME: LLM-assisted hierarchical organization of scientific studies for literature review support. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11… find
  16. Georgios Kontonatsios, Sally Spencer, Peter Matthew, and Ioannis Korkontzelos. 2020. Using a neural network-based feature extraction method to facilitate citation screening for systematic reviews. Expert Systems with Applications: X, 6:100030. find
  17. Xun Liang, Jiawei Yang, Yezhaohui Wang, Chen Tang, Zifan Zheng, Shichao Song, Zehao Lin, Yebin Yang, Simin Niu, Hanyu Wang, Bo Tang, Feiyu Xiong, Keming Mao, and Zhiyu li. 2025. Surveyx: Academic survey automation via large language models. (arXiv:2502.14776). ArXiv:2502.14776 [cs]. — [arXiv:2502.14776](https://arxiv.org/abs/2502.14776) find
  18. Zehan Qi, Rongwu Xu, Zhijiang Guo, Cunxiang Wang, Hao Zhang, and Wei Xu. 2024. Long 2 rag : Evaluatinglong − contextlong − f ormretrieval − augmentedgenerationwithkeypointrecall. In Findings of the Association for Computational find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2025-large-language-models-automated-literature/