Elamur Research
☰
home / papers / 2023 / Can AI-Generated Text be Reliably Detected?

Can AI-Generated Text be Reliably Detected?

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2303.11156

Summary

The paper tests whether current detectors of machine-written text still work when someone deliberately tries to fool them. The authors evaluate four detector families on passages of about 300 tokens: soft watermarking (Kirchenbauer et al.), trained classifiers such as OpenAI's RoBERTa-based GPT-2 detectors, zero-shot methods like DetectGPT, and the retrieval-based detector of Krishna et al.

Their main tool is recursive paraphrasing, in which a neural paraphraser rewrites an LLM's output several times in a row. This sharply cuts detection rates. Text quality drops only modestly, as measured by MTurk raters, perplexity and QA benchmark accuracy.

They also demonstrate spoofing. By querying a watermarked model many times, an attacker can infer its hidden green-list pattern and write human text that gets flagged as AI-generated, which threatens developers' reputation.

Finally, a theorem shows that the highest AUROC any detector can reach depends on how far apart human-written and model-generated text are, measured by Total Variation distance. Experiments indicate this distance shrinks as models grow, so reliable detection should become progressively harder.

Key points

  • Under recursive paraphrasing, detection falls sharply across all four families:
    • watermark TPR@1%FPR drops from 99.3% to 9.7%;
    • DetectGPT AUROC drops from 96.5% to 25.2%;
    • OpenAI's RoBERTa-Large-Detector drops from 100% to 60% TPR@1%FPR;
    • the retrieval detector's accuracy falls from 100% to under 60%.
  • In MTurk evaluation, 77% of recursively paraphrased passages were rated high quality for content preservation and 89% for grammar. QA benchmark accuracy stayed essentially unchanged.
  • Spoofing works without white-box access to the detector. Repeated queries to the watermarked LLM are enough to reconstruct its signature well enough to forge "watermarked" human text.
  • Theorem 1 bounds the best detector's AUROC by a function of TV(M, H):
    • when TV is below 0.2, the best possible AUROC falls under 0.7;
    • a regime with FPR below 0.01 and TPR above 0.9 cannot be guaranteed even at TV = 0.89.
  • Empirical support for shrinking TV comes from two experiments:
    • single-layer LSTMs trained on synthetic Markov data get closer to the true distribution as their size increases;
    • meta-token TV estimates against WebText decrease from GPT-2-Small through GPT-2-XL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

142 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 136 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.11156 · arXiv 2303.11156

@misc{sadasivan2023can,
  title = {{Can AI-Generated Text be Reliably Detected?}},
  author = {Sadasivan, Vinu Sankar and Kumar, Aounon and Balasubramanian, Sriram and Wang, Wenxiao and Feizi, Soheil},
  year = {2023},
  doi = {10.48550/arxiv.2303.11156},
  eprint = {2303.11156},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.11156}
}

References

59 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Scott Aaronson. My ai safety lecture for ut effective altruism. November 2022. URL https://scottaaronson. blog/?p=6823. find
  2. David Ifeoluwa Adelani, Haotian Mai, Fuming Fang, Huy H Nguyen, Junichi Yamagishi, and Isao Echizen. Generating sentiment-preserving fake online reviews using neural language models and their human-and machine-based detection. In Advanced Information Networking and Applications: Proceedings of the 3… find
  3. Noor Al-Sibai. Ai plagiarism detection software keeps falsely accusing students of cheating. Futurism, 2023. URL https://futurism.com/ai-plagiarism-software-false-accusing-students. ↗
  4. Mikhail J Atallah, Victor Raskin, Michael Crogan, Christian Hempelmann, Florian Kerschbaum, Dina Mohamed, and Sanket Naik. Natural language watermarking: Design, analysis, and a proof-of-concept implementation. In Information Hiding: 4th International Workshop, IH 2001 Pittsburgh, PA, USA, April 25–… find
  5. Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and Arthur Szlam. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351, 2019. — [arXiv:1906.03351](https://arxiv.org/abs/1906.03351) find
  6. Lenore Blum, Manuel Blum, and Mike Shub. Comparison of two pseudo-random number generators. In Advances in Cryptology: Proceedings of CRYPTO ’82, pp. 61–78. Plenum, 1982. find
  7. Manuel Blum and Silvio Micali. How to generate cryptographically strong sequences of pseudorandom bits. SIAM Journal on Computing, 13(4):850–864, 1984. doi: 10.1137/0213053. URL https://doi.org/10. 1137/0213053. — [doi:10.1137/0213053](https://doi.org/10.1137/0213053) find
  8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. find
  9. Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. On the possibilities of ai-generated text detection, 2023. find
  10. Jon Christian. Cnet secretly used ai on articles that didn’t disclose that fact, staff say. January 2023. URL https://futurism.com/cnet-ai-articles-label. ↗
  11. Prithiviraj Damodaran. Parrot: Paraphrase generation for nlu., 2021. find
  12. Mehul Reuben Das. Turn-it-in: Ai fails students for not using ai. Firstpost, 2023. URL https://www.firstpost.com/world/ plagiarism-detector-turnitin-keeps-falsely-accusing-students-of-cheating-using-ai-12704662. html. ↗
  13. T Fagni, F Falchi, M Gambini, A Martella, and M Tesconi. Tweepfake: About detecting deepfake tweets. arxiv. arXiv preprint arXiv:2008.00036, 2020. find
  14. Geoffrey A. Fowler. We tested a new chatgpt-detector for teachers. it flagged an innocent student. The Washington Post, 2023. URL https://www.washingtonpost.com/technology/2023/04/01/ chatgpt-cheating-detection-turnitin/. ↗
  15. Henrique Da Silva Gameiro, Andrei Kucharavy, and Ljiljana Dolamic. Llm detectors still fall short of real world: Case of llm-generated short news-like posts, 2024. URL https://arxiv.org/abs/2409.03291. — [arXiv:2409.03291](https://arxiv.org/abs/2409.03291) ↗
  16. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043, 2019. find our card
  17. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. find our card
  18. Chenchen Gu, Xiang Lisa Li, Percy Liang, and Tatsunori Hashimoto. On the learnability of watermarks for language models, 2024. URL https://arxiv.org/abs/2312.04469. — [arXiv:2312.04469](https://arxiv.org/abs/2312.04469) ↗
  19. Kashmir Hill. Accused of cheating by an algorithm, and a professor she had never met. The New York Times, 2022. URL https://www.nytimes.com/2022/05/27/technology/ college-students-cheating-software-honorlock.html. ↗
  20. Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled. arXiv preprint arXiv:1911.00650, 2019. find our card
  21. Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. Automatic detection of machine generated text: A critical survey. arXiv preprint arXiv:2011.01314, 2020. find
  22. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th… find our card
  23. Kafkai. “kafkai: Ai writer & ai content generator”. 2020. URL https://kafkai.com/. ↗
  24. Navid Tafaghodi Khajavi and Anthony Kuh. The quality of the covariance selection through detection problem and AUC bounds. CoRR, abs/1605.05776, 2016. URL http://arxiv.org/abs/1605.05776. — [arXiv:1605.05776](https://arxiv.org/abs/1605.05776) ↗
  25. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint arXiv:2301.10226, 2023a. find our card
  26. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of watermarks for large language models, 2023b. find
  27. Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense, 2023. find our card
  28. Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-free watermarks for language models, 2024. URL https://arxiv.org/abs/2307.15593. ↗
  29. Aounon Kumar, Alexander Levine, Tom Goldstein, and Soheil Feizi. Certifying model accuracy under distribution shifts. arXiv preprint arXiv:2201.12440, 2022. — [arXiv:2201.12440](https://arxiv.org/abs/2201.12440) find
  30. Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. Mage: Machine-generated text detection in the wild, 2024. URL https://arxiv.org/abs/2305. 13242. ↗
  31. Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. Gpt detectors are biased against non-native english writers, 2023. find
  32. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. find
  33. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. find
  34. Ning Lu, Shengcai Liu, Rui He, Qi Wang, Yew-Soon Ong, and Ke Tang. Large language models can be guided to evade ai-generated text detection. arXiv preprint arXiv:2305.10847, 2023. — [arXiv:2305.10847](https://arxiv.org/abs/2305.10847) find
  35. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305, 2023. find our card
  36. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/1808.08745, 2018. find
  37. OpenAI. Gpt-2: 1.5b release. November 2019. URL https://openai.com/research/gpt-2-1-5b-release. ↗
  38. OpenAI. Chatgpt: Optimizing language models for dialogue. November 2022. URL https://openai.com/ blog/chatgpt/. ↗
  39. OpenAI. Gpt-4 technical report. March 2023. URL https://cdn.openai.com/papers/gpt-4.pdf. ↗ our card
  40. J. Pu, Z. Sarwar, S. Abdullah, A. Rehman, Y. Kim, P. Bhattacharya, M. Javed, and B. Viswanath. Deepfake text detection: Limitations and opportunities. In 2023 IEEE Symposium on Security and Privacy (SP), pp. 1613–1630, Los Alamitos, CA, USA, may 2023. IEEE Computer Society. doi: 10.1109/SP46215.2023… — [doi:10.1109/sp46215.2023](https://doi.org/10.1109/sp46215.2023) find
  41. Katyanna Quach. Professor freezes student grades after chatgpt claimed ai wrote their papers. The Register, 2023. URL https://www.theregister.com/2023/05/17/university\_chatgpt\_grades/. ↗
  42. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. find
  43. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019. URL https://arxiv.org/abs/1910.10683. ↗
  44. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, art. arXiv:1606.05250, 2016. find our card
  45. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022. find
  46. Vinu Sankar Sadasivan, Mahdi Soltanolkotabi, and Soheil Feizi. Cuda: Convolution-based unlearnable datasets. arXiv preprint arXiv:2303.04278, 2023. find
  47. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022. find our card
  48. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203, 2019. find our card
  49. Omanshu Thapliyal and Inseok Hwang. Data-driven cyberattack synthesis against network control systems, 2022. find
  50. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. find
  51. Wenxiao Wang, Alexander J Levine, and Soheil Feizi. Improved certified defenses against data poisoning with (deterministic) finite aggregation. In International Conference on Machine Learning, pp. 22769–22783. PMLR, 2022. find
  52. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Alham Fikri Aji, and Preslav Nakov. M4: Multi-generator, multi-domain, and multi-lingual black-box machine-generated text detection, 2023. find
  53. Max Weiss. Deepfake bot submissions to federal public comment websites cannot be distinguished from human submissions. Technology Science, 2019121801, 2019. find
  54. Alex Wilson, Phil Blunsom, and Andrew D Ker. Linguistic steganography on twitter: hierarchical language modeling with manual interaction. In Media Watermarking, Security, and Forensics 2014, volume 9028, pp. 9–25. find
  55. Max Wolff. Attacking neural text detectors. CoRR, abs/2002.11768, 2020. URL https://arxiv.org/abs/ 2002.11768. — [arXiv:2002.11768](https://arxiv.org/abs/2002.11768) ↗
  56. Hanlin Zhang, Benjamin L. Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak. Watermarks in the sand: Impossibility of strong watermarking for generative models, 2024. URL https://arxiv.org/abs/2311.04378. — [arXiv:2311.04378](https://arxiv.org/abs/2311.04378) ↗
  57. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained tr… find
  58. Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for ai-generated text, 2023a. URL https://arxiv.org/abs/2306.17439. ↗
  59. Xuandong Zhao, Yu-Xiang Wang, and Lei Li. Protecting language generation models via invisible watermarking. arXiv preprint arXiv:2302.03162, 2023b. — [arXiv:2302.03162](https://arxiv.org/abs/2302.03162) find