Elamur Research⌕
☰
home / papers / 2026 / Explaining Generalization of AI-Generated Text Detectors Through Linguistic Analysis

Explaining Generalization of AI-Generated Text Detectors Through Linguistic Analysis

Paper · 2026 · Yuxi Xia, Kinga Stańczak and Benjamin Roth

Summary

Fine-tuned classifiers that flag machine-written text score almost perfectly on data resembling their training material. Their accuracy falls, however, when the prompt, the generating model, or the domain changes. Earlier studies documented these drops but rarely explained them.

The authors build a controlled English benchmark. Human texts from four sources (arXiv abstracts, Amazon product reviews, CNN/Daily Mail news, ASQA answers) are paired with counterparts written by 7 LLMs under 6 prompting strategies. They fine-tune XLM-RoBERTa and DeBERTa-V3 detectors on each prompt–model–dataset combination and test them across prompts, models and datasets. They then correlate the resulting accuracies with how much each of 80 linguistic features shifts between training and test data.

Shifts in pronoun frequency, verb tense and active/passive voice are significantly tied to generalization, sometimes strongly. Still, no single cue explains every setting, and cross-domain failures seem driven by wider distributional differences. The study offers an interpretable way to diagnose detector robustness.

Key points

  • The corpus contains 516,000 texts: 12,000 human-written (3,000 per domain) and 504,000 AI-generated.
  • The generators are Mistral-Large-Instruct-2411, DeepSeek-R1-Distill-Llama-70B, Llama-3.3-70B-Instruct, Qwen2.5-72B/-32B/-14B-Instruct and solar-pro-preview-instruct.
  • The six prompting strategies are 0-shot, 3-shot, style imitation, 0-shot CoT, 1-shot CoT and a multi-round self-refine loop.
  • Both the human and AI texts were cleaned of trivial artifacts such as URLs, emojis, stock openers like "Sure!", headings and lists.
  • Cross-dataset transfer is the weakest setting: detectors trained on abstracts drop to about 57% accuracy elsewhere. Some features reach Pearson r > 0.7 in specific configurations.
  • The authors check robustness with Bonferroni and Benjamini–Hochberg FDR corrections plus Spearman correlation:
    • Pronoun, tense and voice effects hold for cross-prompt and cross-model transfer.
    • No individual feature survives correction in the cross-dataset setting.
  • Code and data are released at github.com/Yuuxii/Generalization-of-AI-text-Detector.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ViennaVienna · Austria16 works · 38,729 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Yuxi Xia, Kinga Stańczak and Benjamin Roth. 2026. Explaining Generalization of AI-Generated Text Detectors Through Linguistic Analysis. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.307

@inproceedings{xia2026explaining,
  title = {{Explaining Generalization of AI-Generated Text Detectors Through Linguistic Analysis}},
  author = {Yuxi Xia, Kinga Stańczak and Benjamin Roth},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.307},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.307}
}

References

55 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mervat Abassy, Kareem Elozeiri, Alexander Aziz, Minh Ngoc Ta, Raj Vardhan Tomar, Bimarsha Adhikari, Saad El Dine Ahmed, Yuxia Wang, Osama Mohammed Afzal, Zhuohan Xie, Jonibek Mansurov, Ekaterina Artemova, Vladislav Mikhailov, Rui Xing, Jiahui Geng, Hasan Iqbal, Zain Muhammad Mujahid, Tarek Mahmoud,… find
  2. Khawla Ali Abd Al-Hameed. 2022. Spearman’s correlation coefficient in statistical analysis. International Journal of Nonlinear Analysis and Applications, 13(1):3249–3255. find
  3. Amrita Bhattacharjee, Raha Moraffah, Joshua Garland, and Huan Liu. 2024. Eagle: A domain generalization framework for ai-generated text detection. ArXiv, abs/2403.15690. — [arXiv:2403.15690](https://arxiv.org/abs/2403.15690) find
  4. Małgorzata Bogdan, Jayanta K Ghosh, Surya T Tokdar, et al. 2008. A comparison of the benjamini-hochberg procedure with some bayesian rules for multiple testing. In Beyond parametrics in interdisciplinary research: Festschrift in honor of Professor Pranab K. Sen, volume 1, pages 211–231. Institute of… find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  6. Colin B. Clement, Matthew Bierbaum, Kevin P. O’Keeffe, and Alexander A. Alemi. 2019. On the use of arxiv as a dataset. ArXiv, abs/1905.00075. — [arXiv:1905.00075](https://arxiv.org/abs/1905.00075) find
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  8. Michael A. Covington and Joe D. McFall. 2010. Cutting the gordian knot: The moving-average type–token ratio (mattr). Journal of Quantitative Linguistics, 17(2):94–100. find
  9. DATAtab Team. 2025. Pearson correlation: A beginner’s guide. https://datatab.net/tutorial/ pearson-correlation. DATAtab: Online Statistics Calculator. DATAtab e.U., Graz, Austria. ↗
  10. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  11. Heather Desaire, Aleesa E. Chua, Madeline Isom, Romana Jarosova, and David Hua. 2023. Distinguishing academic science writing from humans or chatgpt with over 99% accuracy using off-the-shelf machine learning tools. Cell Reports Physical Science, 4(6):101426. find
  12. Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. 2024. RAID: A shared benchmark for robust evaluation of machine-generated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui…
  13. Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116, Florence, Italy. Association for Computat…
  14. Georgios P. Georgiou. 2024. Differentiating between human-written and ai-generated texts using linguistic features automatically extracted from an online computational tool. Inf., 16:979. find
  15. Jelle J. Goeman and Aldo Solari. 2014. Multiple hypothesis testing in genomics. Statistics in medicine, 33(11):1946–1978. find
  16. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. ArXiv, abs/2301.07597.
  17. Xun Guo, Yongxin He, Shan Zhang, Ting Zhang, Wanquan Feng, Haibin Huang, and Chongyang Ma. 2024. Detective: Detecting ai-generated text via multi-level contrastive learning. Advances in Neural Information Processing Systems, 37:88320–88347. find
  18. Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Spotting llms with binoculars: zero-shot detection of machine-generated text. In Proceedings of the 41st International Conference on Machine Learning, ICML’2… find
  19. Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient disentangled embedding sharing. ICLR 2023. — DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
  20. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
  21. Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. 2024. Bridging language and items for retrieval and recommendation. ArXiv, abs/2403.03952. — [arXiv:2403.03952](https://arxiv.org/abs/2403.03952) find
  22. Xiaomengc Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2023. Radar: robust ai-text detection via adversarial learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23. find
  23. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22.
  24. Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. 2024. MAGE: Machine-generated text detection in the wild. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 36–53, Bangk…
  25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.
  26. Zeyan Liu, Zijun Yao, Fengjun Li, and Bo Luo. 2024. On the detectability of chatgpt content: Benchmarking, methodology, and evaluation through the lens of academic writing. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24, page 2236–2250, New York,… find
  27. Dominik Macko, Jakub Kopál, Robert Moro, and Ivan Srba. 2025. MultiSocial: Multilingual benchmark of machine-generated text detection of social-media texts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 727–752, Vienna, Aust…
  28. Dominik Macko, Robert Moro, Adaku Uchendu, Jason Samuel Lucas, Michiharu Yamashita, Matúš Pikuliak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, and Maria Bielikova. 2023. MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark. In Proceedings of the 2023 Conference on Empiri… find
  29. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: iterative refinement with… find
  30. Anastasia Margolina and Anastasia Kolmogorova. 2023. Exploring Evaluation Techniques in Controlled Text Generation: A Comparative Study of Semantics and Sentiment in ruGPT3large-Generated and Human-Written Movie Reviews. In COMPUTATIONAL LINGUISTICS AND INTELLECTUAL TECHNOLOGIES”. RSUH. find
  31. David M. Markowitz, Jeffrey T. Hancock, and Jeremy N. Bailenson. 2024. Linguistic markers of inherently false ai communication and intentionally false human communication: Evidence from hotel reviews. Journal of Language and Social Psychology, 43(1):63–82. find
  32. Daniel Mathews, Justin P Varghese, and Libin Chacko Samuel. 2024. Classifying ai-generated summaries and human summaries based on statistical features. In 2024 International Conference on Trends in Quantum Computing and Emerging Business Technologies, pages 1–5. find
  33. Akshay Mendhakar and Darshan H S. 2023. Parts-of-speech (pos) analysis and classification of various text genres. Corpus-based Studies across Humanities, 1(1):99–131. find
  34. Meta AI. 2024. Introducing meta llama 3: The most capable openly available llm to date. Accessed: 2025-05-01. find
  35. Mistral AI. 2024. Mistral large instruct 2411. Accessed: 2025-05-01. find
  36. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. Detectgpt: zero-shot machine-generated text detection using probability curvature. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
  37. Alberto Muñoz-Ortiz, Carlos Gómez-Rodríguez, and David Vilares. 2024. Contrasting linguistic patterns in human and llm-generated news text. Artificial Intelligence Review, 57(10). find
  38. Inez Okulska, Daria Stetsenko, Anna Kołos, Agnieszka Karlińska, Kinga Gł abińska, and Adam Nowakowski. 2023. Stylometrix: An open-source multilingual tool for representing stylometric vectors. Preprint, arXiv:2309.12810. — [arXiv:2309.12810](https://arxiv.org/abs/2309.12810) find
  39. Chidimma Opara. 2024. Styloai: Distinguishing ai-generated content with stylometric analysis. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky, pages 105–114, Cham. Spr… find
  40. Jacques Savoy. 2020. Machine Learning Methods for Stylometry: Authorship Attribution and Author Profiling. Springer International Publishing, Cham. find
  41. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada. Association for Comput… find
  42. Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. ASQA: Factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8273–8288, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  43. Zhen Tao, Zhiyu Li, Dinghao Xi, and Wei Xu. 2024. CUDRT: Benchmarking the Detection of Human vs. Large Language Models Generated Texts. arXiv preprint. ArXiv:2406.09056 [cs]. — [arXiv:2406.09056](https://arxiv.org/abs/2406.09056) find
  44. Upstage. 2024. Solar pro preview instruct. https://huggingface.co/upstage/ solar-pro-preview-instruct. Instruction-tuned 22B-parameter LLM optimized for single-GPU performance. Preview released September 2024. ↗
  45. Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. 2024. Ghostbuster: Detecting text ghostwritten by large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)…
  46. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024a. M4GTbench: Evaluation benchmark for black-box machine-generated text detecti… find
  47. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, Thomas Arnold, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024b. M4: Multi-generator, multi-domain, and multi-lingual blac… find
  48. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22.
  49. Han Xu, Jie Ren, Pengfei He, Shenglai Zeng, Yingqian Cui, Amy Liu, Hui Liu, and Jiliang Tang. 2024. On the generalization of training-based ChatGPT detection methods. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7223–7243, Miami, Florida, USA. Association for Compu… find
  50. Annepaka Yadagiri, Lavanya Shree, Suraiya Parween, Anushka Raj, Shreya Maurya, and Partha Pakray. 2024. Detecting AI-generated text with pre-trained models using linguistic features. In Proceedings of the 21st International Conference on Natural Language Processing (ICON), pages 188–196, AU-KBC Rese… find
  51. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei… find
  52. Hatice Yildiz Durak, Figen Eğin, and Aytuğ Onan. 2025. A Comparison of Human-Written Versus AI-Generated Text in Discussions at Educational Settings: Investigating Features for ChatGPT, Gemini and BingAI. European Journal of Education, 60(1):e70014. find
  53. Peipeng Yu, Jiahan Chen, Xuan Feng, and Zhihua Xia. 2025. Cheat: A large-scale dataset for detecting chatgpt-written abstracts. IEEE Transactions on Big Data, 11(3):898–906. find
  54. Sergio E. Zanotto and Segun Aroyehun. 2024. Human variability vs. machine consistency: A linguistic analysis of texts generated by humans and large language models. Preprint, arXiv:2412.03025. — [arXiv:2412.03025](https://arxiv.org/abs/2412.03025) find
  55. Yuehan Zhang, Yongqiang Ma, Jiawei Liu, Xiaozhong Liu, Xiaofeng Wang, and Wei Lu. 2024. Detection vs. anti-detection: Is text generated by ai detectable? In Wisdom, Well-Being, Win-Win, pages 209–222, Cham. Springer Nature Switzerland. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xia-2026-explaining-generalization-ai-generated-text/