Elamur Research⌕
☰
home / papers / 2024 / Shallow Synthesis of Knowledge in GPT-Generated Texts: A Case Study in Automatic Related Work Composition

Shallow Synthesis of Knowledge in GPT-Generated Texts: A Case Study in Automatic Related Work Composition

Paper · 2024 · Anna Martin-Boyle; Aahan Tyagi; Marti A. Hearst; and Dongyeop Kang

Summary

The authors ask whether GPT-4 can genuinely synthesize prior literature when writing Related Work sections, or whether it only lists sources. They built ScholaCite, a web tool in which GPT-4 proposes groupings of cited papers with rationales and then drafts text that a user can edit. The test bed is ten best papers from ACL 2023. For each paper, the model receives a "work-in-progress" version with the original Related Work section removed, plus the abstracts of the works that section cited.

Three versions are compared: the original human text, a human-AI collaborative (GPT-assisted) text, and a fully GPT-generated one. To score them, the authors propose a citation-graph method. Each cited work is a node, and an edge links two works cited in the same sentence. Graph structure then serves as a reproducible proxy for how well a text connects its sources.

Purely generated sections produce sparse, weakly linked graphs. Collaborative ones look structurally close to human writing. The authors conclude that LLMs can help brainstorm citation groupings but should not draft sections on their own.

Key points

  • Graph density: human-written 0.14 and GPT-assisted 0.13, versus only 0.02 for GPT-generated texts.
  • Average clustering coefficient: human-written 0.66 and GPT-assisted 0.63, versus 0.15 for GPT-generated texts. Human and collaborative texts arrange citations into clearer subgroups.
  • Inputs: the model saw only abstract-level information about cited works. The authors name this and the limited context window as likely reasons for shallow synthesis.
  • Limitations:
    • The sample is small (10 NLP papers).
    • Only one prompt and hyperparameter setup was tested.
    • ScholaCite was used by just one of the authors, so no user study was done.
  • Design recommendation: writing assistants should support repetitive tasks, such as organizing citations, rather than generating full sections of prose.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States175 works · 227,440 citations🎓University of MinnesotaMinneapolis · United States28 works · 2,184 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArts and Humanities · 16,039 worksInformation Systems · 1,437 worksDecision Sciences · 209 worksStatistics, Probability and Uncertainty · 61 worksHistory and Philosophy of Science · 53 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Anna Martin-Boyle; Aahan Tyagi; Marti A. Hearst; and Dongyeop Kang. 2024. Shallow Synthesis of Knowledge in GPT-Generated Texts: A Case Study in Automatic Related Work Composition. arXiv:2402.12255.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.12255 · arXiv 2402.12255

@misc{hearst2024shallow,
  title = {{Shallow Synthesis of Knowledge in GPT-Generated Texts: A Case Study in Automatic Related Work Composition}},
  author = {Anna Martin-Boyle; Aahan Tyagi; Marti A. Hearst; and Dongyeop Kang},
  year = {2024},
  doi = {10.48550/arxiv.2402.12255},
  eprint = {2402.12255},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.12255}
}

References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed AbuRa’ed, Horacio Saggion, Alexander Shvets, and Àlex Bravo. 2020. Automatic related work section generation: experiments in scientific document abstracting. Scientometrics, 125(3):3159–3185. find
  2. Ekin Akyurek and Jacob Andreas. 2023. LexSym: Compositionality as lexical symmetry. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 639–657, Toronto, Canada. Association for Computational Linguistics. find
  3. Adhari AlZaabi, Amira ALAmri, Halima Albalushi, Ruqaya Aljabri, and AbdulRahman AalAbdulsalam. 2023. Chatgpt applications in Academic Research: A Review of Benefits, Concerns, and Recommendations. find
  4. Lutz Bornmann, Robin Haunschild, and Rüdiger Mutz. 2021. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications, 8(1):224. find
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  6. Oğuz ’Oz’ Buruk. 2023. Academic Writing with GPT-3.5: Reflections on Practices, Efficacy and Transparency. arXiv. find
  7. Carnegie Mellon University. Literature reviews. Carnegie Mellon University Student Success Resources. Accessed: 2024-01-16. find
  8. Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Rui Yan, Xin Gao, and Xiangliang Zhang. 2022. Target-aware Abstractive Related Work Generation with Contrastive Learning. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM. find
  9. Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Xiangliang Zhang, Dongyan Zhao, and Rui Yan. 2021. Capturing Relations between Scientific Papers: An Abstractive Model for Related Work Section Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and… find
  10. Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023. Marked personas: Using natural language prompts to measure stereotypes in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1504–1532, Toronto, Canada. Associatio…
  11. Julia Colyar. 2009. Becoming writing, becoming writers. Qualitative Inquiry, 15(2):421–436. find
  12. Richeek Das, Sahasra Ranjan, Shreya Pathak, and Preethi Jyothi. 2023. Improving pretraining techniques for code-switched NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1176–1191, Toronto, Canada. Association for Computat… find
  13. Patrick Fernandes, Kayo Yin, Emmy Liu, André Martins, and Graham Neubig. 2023. When does translation require context? a data-driven, multilingual exploration. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 606–626, Toronto, C… find
  14. Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. 2023. Do androids laugh at electric sheep? humor “understanding” benchmarks from the new yorker caption contest. In Proceedings of the 61st Annual Meeting of the Association for Computatio…
  15. Cong Duy Vu Hoang and Min-Yen Kan. 2010. Towards Automated Related Work Summarization. COLING. find
  16. Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze. 2023. Glot500: Scaling multilingual corpora and language models to 500 languages. In Proceedings of the 61st Annu… find
  17. David Jurgens, Srijan Kumar, Raine Hoover, Dan Mc-Farland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391–406.
  18. Army Justitia and Hei-Chia Wang. 2022. Automatic Related Work Section in Scientific Article: Research Trends and Future Directions. In 2022 International Seminar on Intelligent Technology and Its Applications (ISITIA). IEEE. find
  19. Turgut Karakose. 2023. The Utility of ChatGPT in Educational Research—Potential Opportunities and Pitfalls. Educational Process International Journal, 12(2). find
  20. Mario Krenn, Lorenzo Buffoni, Bruno Coutinho, Sagi Eppel, Jacob Gates Foster, Andrew Gritsevskiy, Harlin Lee, Yichao Lu, João P. Moutinho, Nima Sanjabi, Rishi Sonthalia, Ngoc Mai Tran, Francisco Valente, Yangxinyu Xie, Rose Yu, and Michael Kopp. 2023. Forecasting the future of artificial intelligenc… find
  21. Xiangci Li and Jessica Ouyang. 2022. Automatic Related Work Generation: A Meta Study. arXiv.org. find
  22. Ziqiao Ma, Jiayi Pan, and Joyce Chai. 2023. World-to-words: Grounded open vocabulary acquisition through fast mapping in vision-language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 524–544, Toronto, Canada. Associa… find
  23. Jean Maillard, Cynthia Gao, Elahe Kalbassi, Kaushik Ram Sadagopan, Vedanuj Goswami, Philipp Koehn, Angela Fan, and Francisco Guzman. 2023. Small data, big impact: Leveraging minimal data for effective machine translation. In Proceedings of the 61st Annual Meeting of the Association for Computational… find
  24. Mizanur Rahman, Harold Jan R Terano, Nafizur Rahman, Aidin Salamzadeh, and Saidur Rahaman. 2023. find
  25. Chatgpt and Academic Research: A Review and Recommendations Based on Practical Examples. Journal of Education, Management and Development Studies, 3(1):1–12. find
  26. Nada Sahlab, Hesham Kahoul, Nasser Jazdi, and Michael Weyrich. 2022. A Knowledge Graph-Based Method for Automating Systematic Literature Reviews. find
  27. Jaromir Savelka, Kevin D. Ashley, Morgan A Gray, Hannes Westermann, and Huihui Xu. 2023. Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise? arXiv. find
  28. Darsh Shah and Regina Barzilay. 2021. Generating Related Work. arXiv.org. find
  29. Eugênio Monteiro da Silva Júnior and Moisés Lima Dutra. 2021. A roadmap toward the automatic composition of systematic literature reviews. Iberoamerican Journal of Science Measurement and Communication, 1(2):1–22. find
  30. Christoph Tauchert, Marco Bender, Neda Mesbah, and Peter Buxmann. 2020. Towards an Integrative Approach for Automated Literature Reviews Using Machine Learning. In Proceedings of the Annual Hawaii International Conference on System Sciences. Hawaii International Conference on System Sciences. find
  31. Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006. An annotation scheme for citation function. In Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue, pages 80–87, Sydney, Australia. Association for Computational Linguistics. find
  32. Pancheng Wang, Shasha Li, Haifang Zhou, Jintao Tang, and Ting Wang. 2020. Toc-RWG: Explore the Combination of Topic Model and Citation Information for Automatic Related Work Generation. IEEE Access, 8:13043–13055. find
  33. Zhuoer Wang, Marcus Collins, Nikhita Vedula, Simone Filice, Shervin Malmasi, and Oleg Rokhlenko. 2023. Faithful low-resource data-to-text generation through cycle training. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2847–… find
  34. Weiqi Wu, Chengyue Jiang, Yong Jiang, Pengjun Xie, and Kewei Tu. 2023. Do PLMs know and understand ontological knowledge? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3080–3101, Toronto, Canada. Association for Computationa… find
  35. Zekun Deng, Zixin Zeng, Weiye Gu, Jiawen Ji, and Bolin Hua. 2021. Automatic Related Work Section Generation by Sentence Extraction and Reordering. AII@iConference. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/martin-boyle-2024-shallow-synthesis-knowledge-gpt-generated/