Elamur Research⌕
☰
home / papers / 2023 / RADAR: Robust AI-Text Detection via Adversarial Learning

RADAR: Robust AI-Text Detection via Adversarial Learning

Paper · 2023 · Xiaomeng Hu, Pin-Yu Chen and Tsung-Yi Ho

Summary

Detectors that flag text written by large language models break down easily when the AI output is rewritten by another LLM. This failure matters because it enables fake content and plagiarism, and it can also lead to honest authors being wrongly accused. The authors propose RADAR, which borrows the two-player structure of GANs. A tunable paraphraser, initialised from T5-large, tries to rewrite AI-generated text so that it passes as human. A separate tunable detector learns to separate human text from both the original and the rewritten machine text. The paraphraser is updated with PPO, using the detector's score as its reward. The detector is trained with a logistic loss. The two models alternate updates until their validation losses level off.

The method was tested on texts from 8 open LLMs across 4 human-text datasets and compared against 6 existing detectors. RADAR performs about as well as the baselines on unmodified AI text. When the text is rewritten by GPT-3.5-Turbo, a paraphraser never seen during training, RADAR is far more robust than the baselines. Detectors trained this way also transfer well to other generators, which points toward a general-purpose detector.

Key points

  • Eight models served as targets: LLaMA, Vicuna, Pythia, GPT-J, Camel, Palmyra, and both Dolly 1.0 and Dolly 2.0. Against the unseen GPT-3.5-Turbo paraphraser, RADAR's AUROC was 31.64% higher than the strongest existing detector.
  • Transfer works across models. A detector trained on Camel-5B outputs scored 0.915 AUROC on GPT-4 generations. Detectors built from instruction-tuned models such as Vicuna-7B also generalised to other LLMs.
  • On Xsum, RADAR was the only detector that stayed robust under repeated rounds of GPT-3.5-Turbo paraphrasing. It also remained robust against the Dipper paraphraser.
  • The adversarially trained paraphraser improved over its T5-large starting point, both in GPT-3.5-Turbo ratings and in iBLEU on Quora Question Pairs (QQP).
  • On non-paraphrased text, RADAR can trail the best baseline slightly. Adjusting the loss weight λ from 0.5 to 0.6 (Vicuna-7B target) raised AUROC on unmodified text from 0.906 to 0.937 and on paraphrased text from 0.892 to 0.920.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

Cite

Cite as Xiaomeng Hu, Pin-Yu Chen and Tsung-Yi Ho. 2023. RADAR: Robust AI-Text Detection via Adversarial Learning. arXiv:2307.03838.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.03838 · arXiv 2307.03838

@misc{hu2023radar,
  title = {{RADAR: Robust AI-Text Detection via Adversarial Learning}},
  author = {Xiaomeng Hu, Pin-Yu Chen and Tsung-Yi Ho},
  year = {2023},
  doi = {10.48550/arxiv.2307.03838},
  eprint = {2307.03838},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.03838}
}

References

37 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jack Brassil, Steven H. Low, Nicholas F. Maxemchuk, and Lawrence O’Gorman. Electronic marking and identification techniques to discourage document copying. IEEE J. Sel. Areas Commun., 13(8):1495–1504, 1995. find
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  3. Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. On the possibilities of ai-generated text detection. CoRR, abs/2304.04736, 2023. — [arXiv:2304.04736](https://arxiv.org/abs/2304.04736) find
  4. Minhao Cheng, Jinfeng Yi, Pin-Yu Chen, Huan Zhang, and Cho-Jui Hsieh. Seq2sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3601–3608, 2020. find
  5. Pedro Colon-Hernandez, Henry Lieberman, Yida Xin, Claire Yin, Cynthia Breazeal, and Peter Chin. Adversarial transformer language models for contextual commonsense inference. CoRR, abs/2302.05406, 2023. — [arXiv:2302.05406](https://arxiv.org/abs/2302.05406) find
  6. Long Dai, Jiarong Mao, Xuefeng Fan, and Xiaoyi Zhou. Deephider: A multi-module and invisibility watermarking scheme for language model. CoRR, abs/2208.04676, 2022. — [arXiv:2208.04676](https://arxiv.org/abs/2208.04676) find
  7. Cyprien de Masson d’Autume, Shakir Mohamed, Mihaela Rosca, and Jack W. Rae. Training language gans from scratch. In NeurIPS, pages 4302–4313, 2019. find
  8. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. GLTR: statistical detection and visualization of generated text. In ACL, pages 111–116, 2019.
  9. Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019. ↗
  10. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020. find
  11. Tom Hosking and Mirella Lapata. Factorising meaning and form for intent-preserving paraphrasing. In ACL/IJCNLP, pages 1405–1418, 2021. find
  12. Tom Hosking, Hao Tang, and Mirella Lapata. Hierarchical sketch induction for paraphrase generation. In ACL, pages 2489–2501, 2022. find
  13. Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled. In ACL, pages 1808–1822, 2020.
  14. Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2017.
  15. Mohan S. Kankanhalli and K. F. Hau. Watermarking of electronic text documents. Electron. Commer. Res., 2(1-2):169–187, 2002. find
  16. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. CoRR, abs/2301.10226, 2023.
  17. Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. CoRR, abs/2303.13408, 2023.
  18. Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. CoRR, abs/2303.13408, 2023.
  19. Thomas Lavergne, Tanguy Urvoy, and François Yvon. Detecting fake content with relative entropy scoring. In Proceedings of the ECAI’08 Workshop on Uncovering Plagiarism, Authorship and Social Software Misuse, Patras, Greece, July 22, 2008, volume 377 of CEUR Workshop Proceedings, 2008. find
  20. Qi Lei, Lingfei Wu, Pin-Yu Chen, Alex Dimakis, Inderjit S Dhillon, and Michael J Witbrock. Discrete adversarial attacks and submodular optimization with applications to text classification. Proceedings of Machine Learning and Systems, 1:146–165, 2019. find
  21. Weixin Liang, Mert Yüksekgönül, Yining Mao, Eric Wu, and James Zou. GPT detectors are biased against non-native english writers. CoRR, abs/2304.02819, 2023.
  22. Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In ICLR, 2017.
  23. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature. CoRR, abs/2301.11305, 2023.
  24. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Train… — Training language models to follow instructions with human feedback
  25. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
  26. Juan Rodriguez, Todd Hay, David Gros, Zain Shamsi, and Ravi Srinivasan. Cross-domain detection of gpt-2-generated technical text. In NAACL, pages 1213–1233, 2022. find
  27. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? CoRR, abs/2303.11156, 2023.
  28. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. find
  29. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203, 2019.
  30. Hong Sun and Ming Zhou. Joint learning of a dual SMT system for paraphrase generation. In ACL, pages 38–42, 2012. find
  31. Umut Topkara, Mercan Topkara, and Mikhail J. Atallah. The hiding virtues of ambiguity: quantifiably resilient watermarking of natural language text through synonym substitutions. In MM&Sec, pages 164–174, 2006. find
  32. Honai Ueoka, Yugo Murawaki, and Sadao Kurohashi. Frustratingly easy edit-based linguistic steganography with a masked language model. In NAACL, pages 5486–5492, 2021. find
  33. Qingyang Wu, Lei Li, and Zhou Yu. Textgail: Generative adversarial imitation learning for text generation. In AAAI, pages 14067–14075, 2021. find
  34. Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. Diversity-promoting GAN: A cross-entropy based generative adversarial network for diversified text generation. In EMNLP, pages 3940–3949, 2018. find
  35. Zichao Yang, Zhiting Hu, Chris Dyer, Eric P. Xing, and Taylor Berg-Kirkpatrick. Unsupervised text style transfer using language models as discriminators. In NeurIPS, pages 7298–7309, 2018.
  36. Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI, pages 2852–2858, 2017. find
  37. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In NeurIPS, pages 9051–9062, 2019.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hu-2023-radar-robust-ai-text-detection/