Elamur Research⌕
☰
home / papers / 2026 / Spotlights and Blindspots: Evaluating Machine-Generated Text Detection

Spotlights and Blindspots: Evaluating Machine-Generated Text Detection

Summary

Detectors of machine-generated text are hard to compare because published systems use different benchmarks, metrics, and evaluation protocols. Stowe and Patil (Pindrop) run a controlled comparison on English data. They test 15 public detector variants from six systems: Binoculars, Fast-DetectGPT, Zippy, RADAR, BiScope, and DeTeCtive. They also test seven models they trained themselves: five fine-tuned transformers and two feature-based classifiers. The test material is seven balanced test sets drawn from MAGE, RAID, H3C+ and M4GT-Bench, plus three creative human-written collections.

No detector wins everywhere. Most do well on some test sets and fail on others. Rankings change sharply depending on which metric is used and where the decision threshold is set. Class imbalance in the test data can make a model look much better or worse than it is. Most systems also wrongly flag a sizeable share of genuinely human texts.

The study matters because detectors already influence high-stakes decisions such as academic grading, where false accusations cause real harm. The authors therefore ask for diverse evaluation data and for metrics that are clearly justified and suited to the intended use.

Key points

  • Systems evaluated:
    • Zero-shot detectors: Binoculars, Fast-DetectGPT (gpt-neo, gpt-j and falcon-7b variants) and Zippy (LZMA and Ensemble).
    • Pre-trained public detectors: RADAR, BiScope (four variants) and DeTeCtive (four variants).
    • Fine-tuned transformers: DistilBERT, BERT, RoBERTa, Longformer and DeBERTa.
    • Feature-based classifiers: an n-gram gradient-boosting model ("mcgovern") and a stylometric ensemble ("stylo").
  • Test data:
    • MAGE contributes three test sets: mage, mage-ood and mage-ood-para.
    • The other four test sets are raid, h3c+ qa, h3c+ si and m4gt.
    • Each set is a class-balanced sample of 10k texts, and the authors checked that none overlaps with any detector's training data.
  • Spread of results: Across four of the datasets, F1 ranges from nearly 0 up to 0.982. Fine-tuning on in-domain data usually beats the public models, though out-of-domain training is better for some datasets.
  • Human-written texts: On the three new human-written collections, all but three variants mislabel at least 15% of texts on one or more sets. The three exceptions achieve this only by having low recall everywhere.
  • Metric sensitivity: Across eight common metrics, the variance of model rankings ranges from 0.77 to 15.25. F1, AUROC and TPR@FPR 1% behave differently when label distributions are imbalanced. Input length, repetition, punctuation and perplexity partly explain why detector families differ.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3mzmq2zqni35

@inproceedings{stowe2026spotlights,
  title = {{Spotlights and Blindspots: Evaluating Machine-Generated Text Detection}},
  author = {Stowe, Kevin and Patil, Kailash},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3mzmq2zqni35},
  url = {https://doi.org/10.63317/3mzmq2zqni35}
}

References

51 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. The falcon series of open language mo… find our card
  2. Tal August, Maarten Sap, Elizabeth Clark, Katharina Reinecke, and Noah A. Smith. 2020. Exploring the effect of author and reader identity in online story writing: the STORIESINTHEWILD corpus. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 46–54,… find
  3. Guangsheng Bao, Yanbin Zhao, Juncai He, and Yue Zhang. 2025. Glimpse: Enabling white-box methods to use proprietary models for zero-shot llm-generated text detection. find
  4. Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2024. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. find our card
  5. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. find
  6. Amrita Bhattacharjee, Raha Moraffah, Joshua Garland, and Huan Liu. 2024. Eagle: A domain generalization framework for ai-generated text detection. find
  7. Steven Bird, Edward Loper, and Ewan Klein. 2009. Natural Language Processing with Python. O’Reilly Media Inc. find
  8. Scott A. Crossley, Yu Tian, Perpetual Baffour, Alex Franklin, Meg Benner, and Ulrich Boser. 2024. A large-scale corpus for assessing written argumentation: Persuade 2.0. Assessing Writing, 61:100865. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  10. Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. 2024. RAID: A shared benchmark for robust evaluation of machine-generated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui… find
  11. Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116, Florence, Italy. Association for Computat… find our card
  12. German Gritsai, Anastasia Voznyuk, Andrey Grabovoy, and Yury Chekhovich. 2025. Are ai detectors good enough? a survey on quality of datasets with machine-generated texts. find
  13. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. find our card
  14. Hanxi Guo, Siyuan Cheng, Xiaolong Jin, Zhuo Zhang, Kaiyuan Zhang, Guanhong Tao, Guangyu Shen, and Xiangyu Zhang. 2024a. Biscope: Ai-generated text detection by checking memorization of preceding tokens. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 104065–104090. find
  15. Xun Guo, Shan Zhang, Yongxin He, Ting Zhang, Wanquan Feng, Haibin Huang, and Chongyang Ma. 2024b. Detective: Detecting ai-generated text via multi-level contrastive learning. find
  16. Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Spotting llms with binoculars: zero-shot detection of machine-generated text. In Proceedings of the 41st International Conference on Machine Learning, ICML’2… find
  17. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. find our card
  18. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spacy: Industrial-strength natural language processing in python. find
  19. Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2023. Radar: Robust ai-text detection via adversarial learning. In Advances in Neural Information Processing Systems, volume 36, pages 15077– 15095. Curran Associates, Inc. find
  20. Xiaolei Huang, Linzi Xing, Franck Dernoncourt, and Michael J. Paul. 2020. Multilingual Twitter corpus and baselines for evaluating demographic bias in hate speech recognition. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1440–1448, Marseille, France. European Lan… find
  21. Ryuto Koike, Masahiro Kaneko, and Naoaki Okazaki. 2024. Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples. Proceedings of the AAAI Conference on Artificial Intelligence, 38(19):21258–21266. find
  22. Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. 2023. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, WWW ’23, page 3637–3647, New York, NY, USA. Association for Computing Machinery. find
  23. Jooyoung Lee, Xiaochen Zhu, Georgi Karadzhov, Tom Stafford, Andreas Vlachos, and Dongwon Lee. 2025. Collaborative evaluation of deepfake text with deliberation-enhancing dialogue systems. find
  24. Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. 2024. MAGE: Machine-generated text detection in the wild. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 36–53, Bangk… find
  25. Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023. Openorca: An open dataset of gpt augmented flan reasoning traces. find
  26. Zeyan Liu, Zijun Yao, Fengjun Li, and Bo Luo. 2024. On the detectability of chatgpt content: Benchmarking, methodology, and evaluation through the lens of academic writing. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24, page 2236–2250, New York,… find
  27. Chengzhi Mao, Carl Vondrick, Hao Wang, and Junfeng Yang. 2024. Raidar: generative ai detection via rewriting. find
  28. Hope Elizabeth McGovern, Rickard Stureborg, Yoshi Suhara, and Dimitris Alikaniotis. 2025. Your large language models are leaving fingerprints. In Proceedings of the 1st Workshop on GenAI Content Detection (GenAIDetect), pages 85–95, Abu Dhabi, UAE. International Conference on Computational Linguisti… find
  29. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. Detectgpt: zero-shot machine-generated text detection using probability curvature. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. find our card
  30. Mozilla. 2025. Apollodft. find
  31. Alberto Muñoz-Ortiz, Carlos Gómez-Rodríguez, and David Vilares. 2024. Contrasting linguistic patterns in human and LLM-generated news text. Artificial Intelligence Review, 57(10):265. find
  32. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
  33. Jiameng Pu, Zain Sarwar, Sifat Muhammad Abdullah, Abdullah Rehman, Yoonjin Kim, Parantapa Bhattacharya, Mobin Javed, and Bimal Viswanath. 2023. Deepfake text detection: Limitations and opportunities. In 2023 IEEE Symposium on Security and Privacy (SP), pages 1613– 1630. find
  34. Shushanta Pudasaini, Luis Miralles, David Lillis, and Marisa Llorens Salvador. 2025. Benchmarking AI text detection: Assessing detectors against new datasets, evasion tactics, and enhanced LLMs. In Proceedings of the 1stWorkshop on GenAI Content Detection (GenAIDetect), pages 68–77, Abu Dhabi, UAE.… find
  35. Cosmina-Mihaela Rosca, Adrian Stancu, and Emilian Marian Iovanovici. 2025. The new paradigm of deepfake detection at the text level. Applied Sciences, 15(5). find
  36. Sina Mahdipour Saravani, Indrajit Ray, and Indrakshi Ray. 2021. Automated identification of social media bots using deepfake text detection. In Information Systems Security, pages 111–123, Cham. Springer International Publishing. find
  37. Aditya Shah, Prateek Ranka, Urmi Dedhia, Shruti Prasad, Siddhi Muni, and Kiran Bhowmick. 2023. Detecting and unmasking ai-generated texts through explainable artificial intelligence using stylistic features. International Journal of Advanced Computer Science and Applications, 14(10). find
  38. Zhenpeng Su, Xing Wu, Wei Zhou, Guangyuan Ma, and Songlin Hu. 2024. Hc3 plus: A semantic-invariant human chatgpt comparison corpus. find
  39. Thinkst. 2023. Meet zippy: A fast ai/llm text detector. Accessed 2025. find
  40. Adaku Uchendu, Jooyoung Lee, Hua Shen, Thai Le, Ting-Hao ’Kenneth’ Huang, and Dongwon Lee. 2023. Does human collaboration enhance the accuracy of identifying llm-generated deepfake texts? find
  41. Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, and Dongwon Lee. 2021. TURINGBENCH: A benchmark environment for Turing test in the age of neural text generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2001–2016, Punta Cana, Dominican Republic. Association for Com… find
  42. Saranya Venkatraman, Adaku Uchendu, and Dongwon Lee. 2024. GPT-who: An information density-based machine-generated text detector. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 103–115, Mexico City, Mexico. Association for Computational Linguistics. find
  43. Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. 2024. Ghostbuster: Detecting text ghostwritten by large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)… find
  44. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024. M4GT-bench: Evaluation benchmark for black-box machine-generated text detecti… find
  45. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… find our card
  46. Junchao Wu, Runzhe Zhan, Derek F. Wong, Shu Yang, Xinyi Yang, Yulin Yuan, and Lidia S. Chao. 2025. Detectrl: Benchmarking llm-generated text detection in real-world scenarios. find
  47. Kangxi Wu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. 2023. LLMDet: A third party large language models generated text detection tool. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2113–2133, Singapore. Association for Computational Linguistics. find
  48. Xianjun Yang, Wei Cheng, Yue Wu, Linda Petzold, William Yang Wang, and Haifeng Chen. 2023. Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text. find
  49. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trai… find
  50. Yi-Fan Zhang, Zhang Zhang, Liang Wang, Tieniu Tan, and Rong Jin. 2023. Assaying on the robustness of zero-shot machine-generated text detectors. find
  51. Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218–1227, Huhhot, China. Chinese Information Processing Society of China. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/stowe-2026-spotlights-blindspots-evaluating-machine-generated/