Elamur Research⌕
☰
home / papers / 2025 / ChatCite: LLM Agent with Human Workflow Guidance for Comparative Literature Summary

ChatCite: LLM Agent with Human Workflow Guidance for Comparative Literature Summary

Paper · 2025 · Yutong Li, Lu Chen, Aiwei Liu et al.

Summary

Writing the comparative part of a literature review is slow, and earlier LLM-based review tools mostly targeted retrieval and screening. They relied on plain Chain-of-Thought prompting for the summary itself, which tends to drop key details, compare papers poorly and describe each work in isolation. This paper treats summarization as a separate task. Given a description of the proposed work and a set of reference papers, the goal is a related-work-style summary. The authors present ChatCite, a training-free agent that follows how human researchers work. First, a Key Element Extractor condenses each paper into structured facets. Then a Reflective Incremental Generator adds papers one at a time, writing comparative drafts while an LLM evaluator votes to keep the strongest candidates. To evaluate output, they also introduce G-Score, an LLM-based metric covering six quality dimensions that tracks human judgments better than ROUGE. With GPT-3.5, ChatCite beats few-shot prompting and its own ablations, and human annotators clearly prefer it. This suggests that workflow-guided LLMs can handle complex multi-document synthesis.

Key points

  • The extractor pulls seven elements from each paper: research questions, methodology, results, conclusions, contributions, innovations and limitations.
  • G-Score rates six dimensions: Consistency, Coherence, Comparative, Integrity, Fluency and Cite Accuracy. Its scores matched the distribution of human ratings.
  • In the ablation study, ChatCite reached a G-Score of about 4.11 out of 5, against 3.24 for GPT-3.5 few-shot. However, ROUGE-1 barely differed (25.30 vs 25.84), which shows ROUGE's weakness for this task.
  • In a human study on 10 samples, ChatCite received the largest share of preference votes. Without incremental generation, summaries became disjointed per-paper descriptions.
  • The evaluation data covers only computer science papers, and outputs remain somewhat unstable. Code is available at github.com/miaisamelia/ChatCite.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksDecision Sciences · 209 worksStatistics, Probability and Uncertainty · 61 works

Cite

Cite as Yutong Li, Lu Chen, Aiwei Liu, Kai Yu and Lijie Wen. 2025. ChatCite: LLM Agent with Human Workflow Guidance for Comparative Literature Summary. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{li2025chatcite,
  title = {{ChatCite: LLM Agent with Human Workflow Guidance for Comparative Literature Summary}},
  author = {Yutong Li, Lu Chen, Aiwei Liu, Kai Yu and Lijie Wen},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.244/}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed AbuRa’ed, Horacio Saggion, Alexander Shvets, and Àlex Bravo. 2020. Automatic related work section generation: experiments in scientific document abstracting. 125(3). find
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  3. Shubham Agarwal, Issam H Laradji, Laurent Charlin, and Christopher Pal. 2024. Litllm: A toolkit for scientific literature review. arXiv preprint arXiv:2402.01788. find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  5. Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Xiangliang Zhang, Dongyan Zhao, and Rui Yan. 2021. Capturing relations between scientific papers: An abstractive model for related work section generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and… find
  6. Xiuying Chen, Hind Alamro, Li Mingzhe, Shen Gao, Rui Yan, Xin Gao, and Xiangliang Zhang. 2022. Target-aware abstractive related work generation with contrastive learning. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. find
  7. Yubin Ge, Ly Dinh, Xiaofeng Liu, Jinsong Su, Ziyao Lu, Ante Wang, and Jana Diesner. 2021. BACO: A background knowledge- and content-based framework for citing sentence generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Jo… find
  8. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356. find
  9. Cong Duy Vu Hoang and Min-Yen Kan. 2010. Towards automated related work summarization20. In Coling 2010: Posters, pages 427–435, Beijing, China. find
  10. Coling 2010 Organizing Committee. find
  11. Jingshan Huang and Ming Tan. 2023. The role of chatgpt in scientific communication: writing better scientific review articles. American Journal of Cancer Research, 13(4):1148. find
  12. Army Justitia and Hei-Chia Wang. 2022. Automatic related work section in scientific article: Research trends and future directions. In 2022 International Seminar on Intelligent Technology and Its Applications (ISITIA), pages 108–114. IEEE. find
  13. Chin-Yew Lin. 2004a. Rouge: A package for automatic evaluation of summaries. In Annual Meeting of the Association for Computational Linguistics.
  14. Chin-Yew Lin. 2004b. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  15. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634.
  16. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  17. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  18. Pancheng Wang, Shasha Li, Haifang Zhou, Jintao Tang, and Ting Wang. 2020. Toc-rwg: Explore the combination of topic model and citation information for automatic related work generation. IEEE Access, 8:13043–13055. find
  19. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  20. Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020. Automatic generation of citation texts in scholarly papers: A pilot study. In Annual Meeting of the Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2025-chatcite-llm-agent-human-workflow/