Elamur Research⌕
☰
home / papers / 2026 / What Are LLMs Doing to Scientific Communication? Measuring Changes in Writing Practices and Reading Experience

What Are LLMs Doing to Scientific Communication? Measuring Changes in Writing Practices and Reading Experience

Summary

This paper asks whether the growing use of LLMs as writing assistants has changed how NLP research is written and how it is read. Prior studies mostly contrasted fully human with fully machine-generated text. The authors instead model LLMs as editors of human drafts. They compare ACL Anthology papers from before ChatGPT's release with papers from after it. They also build a controlled synthetic set in which an LLM polishes passages taken from pre-ChatGPT papers.

The analysis has three layers:

  • Lexical: frequency statistics plus type- and token-level embeddings to trace emerging words and shifts in their meaning.
  • Stylistic: regression over features such as length, sentiment, readability and lexical and grammatical variability.
  • Reading experience: a pilot rating study with domain experts.

Post-ChatGPT and LLM-polished texts favour fancier vocabulary (e.g., enhance over improve), longer and more complex words, particular syntactic patterns and lower lexical diversity. Some words narrow in meaning while others broaden. Experts judged the polished versions clearer and more engaging, yet voiced scepticism about LLMs. The work connects measurable style change to how readers subjectively perceive scientific writing.

Key points

  • Naturalistic corpus: 37,760 papers, about 1.63M paragraphs and 203.8M tokens, split into two periods:
    • t1: 2020–2022, 20,259 papers, ending with EMNLP 2022.
    • t2: July 2023–December 2024, 17,501 papers, starting with ACL 2023.
  • Updated ACL-OCL corpus: extended to roughly 99k papers through the end of 2024. Missing papers were recovered via the Anthology's BibTeX export and text was extracted with GROBID. A one-line script ingests future papers.
  • Synthetic resource: 3,000 pairs of human passages and their LLM-improved versions, of which 200 pairs carry reading-experience annotations.
  • Rating study: 20 NLP experts compared paired texts on four dimensions: clarity, authenticity, trustworthiness and excitement.
  • Main finding: the synthetic data confirms that the stylistic shifts seen in the naturalistic corpus can be attributed to LLM use. Data and code are on GitHub (FilipMiletic/ScientificCommunication).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3ai7wig4fhd8

@inproceedings{miletic2026what,
  title = {{What Are LLMs Doing to Scientific Communication? Measuring Changes in Writing Practices and Reading Experience}},
  author = {Miletić, Filip and Falk, Neele},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3ai7wig4fhd8},
  url = {https://doi.org/10.63317/3ai7wig4fhd8}
}

References

46 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mayowa Akinwande, Oluwaseyi Adeliyi, and Toyy-ibat Yussuph. 2024. Decoding ai and human authorship: Nuances revealed through nlp and statistical analysis. International Journal on Cybernetics & Informatics, 13(4):85–103. find
  2. Yutian Chen, Hao Kang, Vivian Zhai, Liangze Li, Rita Singh, and Bhiksha Raj. 2023. Token prediction as implicit classification to identify LLM-generated text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13112–13120, Singapore. Association for Comp… find
  3. Heather Desaire, Aleesa E. Chua, Madeline Isom, Romana Jarosova, and David Hua. 2023. Distinguishing academic science writing from humans or chatgpt with over 99% accuracy using off-the-shelf machine learning tools. Cell Reports Physical Science, 4(6):101426. find
  4. Berin Doru, Christoph Maier, Johanna Sophie Busse, Thomas Lücke, Judith Schönhoff, Elena Enax-Krumova, Steffen Hessler, Maria Berger, and Marianne Tokic. 2025. Detecting artificial intelligence–generated versus human-written medical student essays: Semirandomized controlled study. JMIR Medical Educa… find
  5. Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. 2024. RAID: A shared benchmark for robust evaluation of machine-generated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui… find
  6. Catherine A. Gao, Frederick M. Howard, Nikolay S. Markov, Emma C. Dyer, Siddhi Ramesh, Yuan Luo, and Alexander T. Pearson. 2023. Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. npj Digital Medicine, 6(1). find
  7. Andrew Gray. 2024. ChatGPT "contamination": estimating the prevalence of LLMs in the scholarly literature. arXiv, abs/2403.16887. — [arXiv:2403.16887](https://arxiv.org/abs/2403.16887) find
  8. Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv, abs/2203.05794. find our card
  9. Chinnappa Guggilla, Budhaditya Roy, Trupti Chavan, Abdul Rahman, and Edward Bowen. 2025. AI generated text detection using instruction fine-tuned large language and transformer-based models. arXiv, abs/2507.05157. — [arXiv:2507.05157](https://arxiv.org/abs/2507.05157) find
  10. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is ChatGPT to human experts? Comparison corpus, evaluation, and detection. arXiv, abs/2301.07597. find our card
  11. Hassan Tarek Hakam, Robert Prill, Lisa Korte, Bruno Lovreković, Marko Ostojić, Nikolai Ramadanov, and Felix Muehlensiepen. 2024. Human-written vs AI-generated texts in orthopedic academic literature: Comparative qualitative analysis. JMIR Formative Research, 8:e52164. find
  12. Ahmed Abdeen Hamed and Xindong Wu. 2023. Improving detection of ChatGPT-generated fake science using real publication text: Introducing xFakeBibs a supervised-learning network algorithm. arXiv, abs/2308.11767. — [arXiv:2308.11767](https://arxiv.org/abs/2308.11767) find
  13. Dmitry Kobak, Rita González-Márquez, Emőke-Ágnes Horvát, and Jan Lause. 2025. Delving into LLM-assisted writing in biomedical publications through excess vocabulary. Science Advances, 11(27). find
  14. Daphne Koller, Andrew Beam, Arjun Manrai, Euan Ashley, Xiaoxuan Liu, Judy Gichoya, Chris Holmes, James Zou, Noa Dagan, Tien Y. Wong, David Blumenthal, and Isaac Kohane. 2024. Why we support and encourage the use of large language models in NEJM AI submissions. NEJM AI, 1(1). find
  15. Bruce W. Lee and Jason Lee. 2023. LFTK: Hand-crafted features in computational linguistics. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 1–19, Toronto, Canada. Association for Computational Linguistics. find
  16. Mina Lee, Percy Liang, and Qian Yang. 2022. CoAuthor: Designing a human-AI collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA. Association for Computing Machinery. find
  17. Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. 2024. MAGE: Machine-generated text detection in the wild. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 36–53, Bangk… find
  18. Cong William Lin and Wu Zhu. 2025. Divergent LLM adoption and heterogeneous convergence paths in research writing. arXiv, abs/2504.13629. — [arXiv:2504.13629](https://arxiv.org/abs/2504.13629) find
  19. Yongqiang Ma, Jiawei Liu, Fan Yi, Qikai Cheng, Yong Huang, Wei Lu, and Xiaozhong Liu. 2023. AI vs. human – differentiation analysis of scientific content generation. arXiv, abs/2301.10416. — [arXiv:2301.10416](https://arxiv.org/abs/2301.10416) find
  20. Dominik Macko, Robert Moro, Adaku Uchendu, Jason Lucas, Michiharu Yamashita, Matúš Pikuliak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, and Maria Bielikova. 2023. MULTITuDE: Large-scale multilingual machine-generated text detection benchmark. In Proceedings of the 2023 Conference on Empirical Met… find
  21. Maximilian Maurer. 2026. elfen: A python package for efficient linguistic feature extraction for natural language datasets. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, Rabat, Morocco. Association for Computati… find
  22. Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Workshop Track Proceedings, Scottsdale, Arizona, USA. find
  23. Alberto Muñoz-Ortiz, Carlos Gómez-Rodríguez, and David Vilares. 2024. Contrasting linguistic patterns in human and LLM-generated news text. Artificial Intelligence Review, 57(10). find
  24. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn:… find
  25. Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. 2022. POTATO: The portable text annotation tool. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 327–33… find
  26. Benedicte Pierrejean and Ludovic Tanguy. 2018. Towards qualitative word embeddings evaluation: Measuring neighbors variation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 32– 39, New Orleans, Lo… find
  27. Paul Rayson, Damon Berridge, and Brian Francis. 2004. Extending the cochran rule for the comparison of word frequencies between corpora. In JADT 2004 : 7es Journées internationales d’Analyse statistique des Données Textuelles. find
  28. Paul Rayson and Roger Garside. 2000. Comparing corpora using frequency profiling. In The Workshop on Comparing Corpora, pages 1–6, Hong Kong, China. Association for Computational Linguistics. find
  29. Radim Řehůřek and Petr Sojka. 2010. Software framework for topic modelling with large corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45– 50, Valletta, Malta. ELRA. find
  30. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,… find
  31. Alex Reinhart, Ben Markey, Michael Laudenbach, Kachatad Pantusen, Ronald Yurko, Gordon Weinberg, and David West Brown. 2025. Do LLMs write like humans? Variation in grammatical and rhetorical styles. Proceedings of the National Academy of Sciences, 122(8). find
  32. Aquia Richburg, Calvin Bao, and Marine Carpuat. 2024. Automatic authorship analysis in human-AI collaborative writing. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1845–1855, Torino, Italia. ELRA a… find
  33. Shaurya Rohatgi, Yanxia Qin, Benjamin Aw, Niranjana Unnithan, and Min-Yen Kan. 2023. The ACL OCL corpus: Advancing open science in computational linguistics. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10348–10361, Singapore. Association for Compu… find
  34. Jenna Russell, Marzena Karpinska, and Mohit Iyyer. 2025. People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5342–5373, Vi… find
  35. Eyal Sagi, Stefan Kaufmann, and Brady Clark. 2009. Semantic density analysis: Comparing word meaning across time and phonetic space. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 104–111, Athens, Greece. Association for Computational Linguistics. find
  36. Dominik Schlechtweg. 2023. Human and Computational Measurement of Lexical Semantic Change. Stuttgart, Germany. find
  37. Zhixiong Su, Yichen Wang, Herun Wan, Zhaohan Zhang, and Minnan Luo. 2025. HACo-det: A study towards fine-grained machine-generated text detection under human-AI coauthoring. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2201… find
  38. Nina Tahmasebi, Lars Borin, and Adam Jatowt. 2021. Survey of computational approaches to lexical semantic change. In Nina Tahmasebi, Lars Borin, Adam Jatowt, Yang Xu, and Simon Hengchen, editors, Computational Approaches to Semantic Change, pages 1–91. Language Science Press, Berlin. find
  39. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, Thomas Arnold, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024. M4: Multi-generator, multi-domain, and multi-lingual black… find
  40. Yuxia Wang, Artem Shelmanov, Jonibek Mansurov, Akim Tsvigun, Nizar Habash, Alham Fikri Aji, Ekaterina Artemova, Zhuohan Xie, Jinyan Su, Rui Xing, Iryna Gurevych, and Preslav Nakov. 2025. PAN’25 generative AI detection (task 2): Human-AI collaborative text classification. find
  41. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory e… find our card
  42. Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia Sam Chao, and Derek Fai Wong. 2025. A survey on LLM-generated text detection: Necessity, methods, and future directions. Computational Linguistics, 51(1):275–338. find
  43. Hatice Yildiz Durak, Figen Eğin, and Aytuğ Onan. 2025. A comparison of human-written versus AI-generated text in discussions at educational settings: Investigating features for ChatGPT, Gemini and BingAI. European Journal of Education, 60(1). find
  44. Olga Zamaraeva, Dan Flickinger, Francis Bond, and Carlos Gómez-Rodríguez. 2025. Comparing LLM-generated and human-authored news text using formal syntactic theory. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9041–9060, Vie… find
  45. Sergio E. Zanotto and Segun Aroyehun. 2024. Human variability vs. machine consistency: A linguistic analysis of texts generated by humans and large language models. arXiv, abs/2412.03025. — [arXiv:2412.03025](https://arxiv.org/abs/2412.03025) find
  46. Xuandong Zhao, Sam Gunn, Miranda Christ, Jaiden Fairoze, Andres Fabrega, Nicholas Carlini, Sanjam Garg, Sanghyun Hong, Milad Nasr, Florian Tramèr, Somesh Jha, Lei Li, Yu-Xiang Wang, and Dawn Song. 2025. SoK: Watermarking for AI-generated content. In IEEE Symposium on Security and Privacy, SP 2025, S… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/miletic-2026-what-are-llms-doing-scientific/