Elamur Research
☰
home / papers / 2019 / Automatic Detection of Generated Text is Easiest when Humans are Fooled

Automatic Detection of Generated Text is Easiest when Humans are Fooled

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1911.00650

Summary

The paper asks why people and automatic classifiers disagree about whether a text came from a person or a language model, and how the decoding strategy shapes that gap. The authors generated excerpts with GPT-2 Large (774M parameters) using three sampling schemes: untruncated random sampling, top-k (k=40) and nucleus sampling (p=0.96). They paired these with human web text drawn from the same distribution as GPT-2's training data. For each scheme and several excerpt lengths, they fine-tuned BERT-based binary classifiers and also ran evaluations with trained human raters.

The main finding is an inverse relationship. Cutting off the low-probability tail, especially with top-k, makes output more convincing to readers. It also leaves statistical traces, such as a skewed token distribution, that classifiers detect easily. Automatic detectors beat humans in every setting, but they transferred poorly across decoding methods. The authors conclude that judging how human-like generated text is requires both human and automatic detectors.

Key points

  • Per decoding method, the training set contains 250,000 generated excerpts paired with 250,000 webtext excerpts from the gpt-2-output-dataset. An extra 5,000 pairs are held out for validation and testing, and excerpts shorter than 192 WordPiece tokens are removed.
  • Top-k output is the hardest for raters but the easiest for classifiers. Raters found untruncated and nucleus (p=0.96) samples equally hard, suggesting that p values well below the 0.92–0.98 range would be needed to seem noticeably more human.
  • Human rater accuracy has a median of 74%, which is below the best discriminator. Even multi-sentence excerpts fooled expert raters more than 30% of the time.
  • Random pairs of raters agreed on only 59% of excerpts, while raters and discriminators agreed on 61–70%. Humans tend to notice semantic errors such as topic drift and falsehoods, whereas classifiers pick up statistical artifacts.
  • Classifiers trained on one strategy generalize poorly to another, but training on mixed data helps. Accuracy grows with excerpt length for both humans and machines. All experiments use English only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

162 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 153 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1911.00650 · arXiv 1911.00650

@misc{ippolito2019automatic,
  title = {{Automatic Detection of Generated Text is Easiest when Humans are Fooled}},
  author = {Ippolito, Daphne and Duckworth, Daniel and Callison-Burch, Chris and Eck, Douglas},
  year = {2019},
  doi = {10.48550/arxiv.1911.00650},
  eprint = {1911.00650},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1911.00650}
}

References

34 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Haotian Mai, Fuming Fang, Huy H Nguyen, Junichi Yamagishi, and Isao Echizen. 2020. Generating sentiment-preserving fake online reviews using neural language models and their human-and machine-based detection. In International Conference on Advanced Information Networking and… find
  2. Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Journal of economic perspectives, 31(2):211–36. find
  3. Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and Arthur Szlam. 2019. find
  4. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351. — [arXiv:1906.03351](https://arxiv.org/abs/1906.03351) find
  5. Nicole A Cooke. 2018. Fake news and alternative facts: Information literacy in a post-truth era. American Library Association. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  7. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898. find
  8. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019. Gltr: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116. find our card
  9. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. find
  10. Anjuli Kannan and Oriol Vinyals. 2017. Adversarial evaluation of dialogue models. arXiv preprint arXiv:1701.08198. — [arXiv:1701.08198](https://arxiv.org/abs/1701.08198) find
  11. Sarah E Kreps, Miles McCain, and Miles Brundage. 2020. All the news thats fit to fabricate: Ai-generated text as a tool of media misinformation. Social Science Research Network. find
  12. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pages 355–368. find
  13. Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547. find
  14. Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. 2017. Adversarial ranking for language generation. In Advances in Neural Information Processing Systems, pages 3155–3165. find
  15. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In International Conference on Learning Representations. find our card
  16. Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic turing test: Learning to evaluate dialogue responses. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P… find our card
  17. Lesly Miculicich, Marc Marone, and Hany Hassan. 2019. Selecting, planning, and rewriting: A modular approach for data-to-document generation and translation. find
  18. EMNLP-IJCNLP 2019, page 289. find
  19. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658–4664, Florence, Italy. Association for Computational Linguistics. find
  20. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. arXiv preprint arXiv:1707.06875. find
  21. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  22. Tal Schuster, Roei Schuster, Darsh J Shah, and Regina Barzilay. 2019. Are we safe yet? the limitations of distributional features for fake news detection. arXiv preprint arXiv:1908.09805. — [arXiv:1908.09805](https://arxiv.org/abs/1908.09805) find
  23. Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 843–861. find
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725. find
  25. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, and Jasmine Wang. 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203. find our card
  26. Jonghyuk Song, Sangho Lee, and Jong Kim. 2015. Crowdtarget: Target-based detection of crowdturfing in online social networks. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pages 793–804. ACM. find
  27. Alan Turing. 1950. Computing machinery and intelligence-am turing. Mind, 59(236):433. find
  28. Chris J Vargo, Lei Guo, and Michelle A Amazeen. 2018. The agenda-setting power of fake news: A big data analysis of the online media landscape from 2014 to 2016. New media & society, 20(5):2028– 2049. find
  29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  30. Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science, 359(6380):1146–1151. find
  31. Gang Wang, Christo Wilson, Xiaohan Zhao, Yibo Zhu, Manish Mohanlal, Haitao Zheng, and Ben Y Zhao. 2012. Serf and turf: crowdturfing for fun and profit. In Proceedings of the 21st international conference on World Wide Web, pages 679–688. ACM. find
  32. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  33. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. CoRR, abs/1905.12616. find our card
  34. Tianyi Zhang, Varsha Kishore, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find