Elamur Research⌕
☰
home / papers / 2023 / On the Reliability of Watermarks for Large Language Models

On the Reliability of Watermarks for Large Language Models

Paper · 2023 · John Kirchenbauer, Jonas Geiping, Yuxin Wen et al.

Summary

Spotting machine-generated text helps limit spam, bot activity and contamination of future training data. Earlier watermark evaluations, however, mostly assumed the detector receives unmodified model output. This paper tests whether the green-list watermark of Kirchenbauer et al. (2023) survives realistic handling: rewriting by people, paraphrasing by a separate non-watermarked LLM, and pasting short watermarked passages into longer human-written documents.

The authors revisit the whole generation and detection pipeline. They introduce a harder-to-reverse-engineer hashing variant (SelfHash) and detection tests designed for documents that mix watermarked spans with human text. They then attack outputs of LLaMA-7B-based models (including Vicuna) using GPT-3.5, the Dipper paraphraser and a human study.

Paraphrasing weakens the watermark signal but rarely removes it. Rewrites tend to keep n-grams or longer fragments of the source, so detection becomes confident again once enough tokens are available. The authors argue that detectors should be compared by how much text they need to succeed. On that measure, watermarking scales better than loss-based detection or retrieval.

Key points

  • Machine paraphrasing: Under attack by GPT-3.5 and Dipper, detection ROC-AUC stays above 0.85 at T = 200 tokens and above 0.9 at T = 600 tokens.
  • Copy-paste setting: When 150 watermarked tokens are inserted into a 600-token human-written passage, detection AUC exceeds 0.95.
  • Human study: 14 graduate-student annotators rewrote watermarked text, with a $100 gift card offered to the most successful evaders. Even after strong human rewriting, the watermark was typically detectable after about 800 tokens at a 1e−5 false positive rate.
  • Hashing schemes: The paper compares LeftHash (context width h = 1, whose green lists are easier for outsiders to infer) with SelfHash (h = 4, minimum-based seeding, and the token being generated included in the seed).
  • Baselines: Under attack, both DetectGPT and the retrieval-based detector proposed by Krishna et al. (2023) perform worse at longer sequence lengths. DetectGPT is also very costly, taking over 12 hours for roughly 500 samples at T = 1000.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States219 works · 103,487 citations🎓University of Maryland, College ParkCollege Park · United States141 works · 29,174 citations🔬Max Planck Institute for Intelligent SystemsStuttgart · Germany10 works · 1,031 citations🔬ELLIS Institute TübingenTübingen · Germany4 works · 226 citations

Also on the byline: Scuola Normale Superiore.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum and Tom Goldstein. 2023. On the Reliability of Watermarks for Large Language Models. arXiv:2306.04634.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.04634 · arXiv 2306.04634

@misc{kirchenbauer2023reliability,
  title = {{On the Reliability of Watermarks for Large Language Models}},
  author = {John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum and Tom Goldstein},
  year = {2023},
  doi = {10.48550/arxiv.2306.04634},
  eprint = {2306.04634},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.04634}
}

References

59 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sahar Abdelnabi and Mario Fritz. Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 121– 140, May 2021. doi: 10.1109/SP40001.2021.00083. — [doi:10.1109/sp40001.2021.00083](https://doi.org/10.1109/sp40001.2021.00083) find
  2. Mikhail J. Atallah, Victor Raskin, Michael Crogan, Christian Hempelmann, Florian Kerschbaum, Dina Mohamed, and Sanket Naik. Natural Language Watermarking: Design, Analysis, and a Proof-of-Concept Implementation. In Ira S. Moskowitz (ed.), Information Hiding, Lecture Notes in Computer Science, pp. 18… — [doi:10.1007/3-540-45496-9](https://doi.org/10.1007/3-540-45496-9) find
  3. Sameer Badaskar, Sachin Agarwal, and Shilpa Arora. Identifying real or fake articles: Towards better language modeling. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-II, 2008. find
  4. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pp. 610–623, New York, NY, USA, March 2021. Ass… — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) find
  5. Daria Beresneva. Computer-generated text detection using machine learning: A systematic review. In 21st International Conference on Applications of Natural Language to Information Systems, NLDB, pp. 421–426. Springer, 2016. find
  6. Meghana Moorthy Bhat and Srinivasan Parthasarathy. How Effectively Can Machines Defend Against Machine-Generated Fake News? An Empirical Study. In Proceedings of the First Workshop on Insights from Negative Results in NLP, pp. 48–53, Online, November 2020. Association for Computational Linguistics.… find
  7. James V. Bradley. Distribution-free Statistical Tests. Fort Belvoir, VA, August 1960. Defense Technical Information Center. doi: 10.21236/AD0249268. URL http://www.dtic.mil/docs/citations/AD0249268. — [doi:10.21236/ad0249268](https://doi.org/10.21236/ad0249268) ↗
  8. Jack T Brassil, Steven Low, Nicholas F Maxemchuk, and Lawrence O’Gorman. Electronic marking and identification techniques to discourage document copying. IEEE Journal on Selected Areas in Communications, 13(8):1495–1504, 1995. find
  9. Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. On the Possibilities of AI-Generated Text Detection. arxiv:2304.04736[cs], April 2023. doi: 10.48550/arXiv.2304.04736. URL http://arxiv.org/abs/2304.04736. — [doi:10.48550/arxiv.2304.04736](https://doi.org/10.48550/arxiv.2304.04736) ↗
  10. Yuei-Lin Chiang, Lu-Ping Chang, Wen-Tai Hsieh, and Wen-Chih Chen. Natural Language Watermarking Using Semantic Substitution for Chinese Text. In Ton Kalker, Ingemar Cox, and Yong Man Ro (eds.), Digital Watermarking, Lecture Notes in Computer Science, pp. 129–140, Berlin, Heidelberg, 2004. Springer.… — [doi:10.1007/978-3-540-24624-4](https://doi.org/10.1007/978-3-540-24624-4) find
  11. William G Cochran. The χ2 test of goodness of fit. The Annals of mathematical statistics, pp. 315–345, 1952. find
  12. Evan Crothers, Nathalie Japkowicz, and Herna Viktor. Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods. arxiv:2210.07321[cs], November 2022. doi: 10.48550/arXiv.2210.07321. URL http://arxiv.org/abs/2210.07321. ↗
  13. Falcon Dai and Zheng Cai. Towards Near-imperceptible Steganographic Text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4303–4308, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1422. URL https://aclanthol… find
  14. Tina Fang, Martin Jaggi, and Katerina Argyraki. Generating Steganographic Text with LSTMs. In Proceedings of ACL 2017, Student Research Workshop, pp. 100–106, Vancouver, Canada, July 2017. Association for Computational Linguistics. URL https://aclanthology.org/P17-3017. ↗
  15. Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. Three bricks to consolidate watermarks for large language models. arXiv preprint arXiv:2308.00113, 2023. — [arXiv:2308.00113](https://arxiv.org/abs/2308.00113) find
  16. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027 [cs], December 2020. URL http://arxiv.org/abs/2… ↗
  17. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043, 2019.
  18. EA Grechnikov, GG Gusev, AA Kustarev, and AM Raigorodsky. Detection of artificial texts. In RCDL2009 Proceedings, pp. 306–308, 2009. find
  19. Alexei Grinbaum and Laurynas Adomaitis. The Ethical Need for Watermarks in Machine-Generated Language. arxiv:2209.03118[cs], September 2022. doi: 10.48550/arXiv.2209.03118. URL http://arxiv.org/abs/2209.03118. — [doi:10.48550/arxiv.2209.03118](https://doi.org/10.48550/arxiv.2209.03118) ↗
  20. Xuanli He, Qiongkai Xu, Lingjuan Lyu, Fangzhao Wu, and Chenguang Wang. Protecting Intellectual Property of Language Generation APIs with Lexical Watermark. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10758–10766, June 2022a. doi: 10.1609/aaai.v36i10.21321. URL ht… — [doi:10.1609/aaai.v36i10.21321](https://doi.org/10.1609/aaai.v36i10.21321) find
  21. Xuanli He, Qiongkai Xu, Yi Zeng, Lingjuan Lyu, Fangzhao Wu, Jiwei Li, and Ruoxi Jia. CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks. In Advances in Neural Information Processing Systems, October 2022b. URL https://openreview.net/forum?id=L7P3IvsoUXY. ↗
  22. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019.
  23. Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. Radar: Robust ai-text detection via adversarial learning. arXiv preprint arXiv:2307.03838, 2023.
  24. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. find
  25. Gabriel Kaptchuk, Tushar M. Jois, Matthew Green, and Aviel Rubin. Meteor: Cryptographically Secure Steganography for Realistic Distributions, 2021. URL https://eprint.iacr.org/2021/686. ↗
  26. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A Watermark for Large Language Models. arxiv:2301.10226[cs], January 2023. doi: 10.48550/arXiv.2301.10226. URL http://arxiv.org/abs/2301.10226. ↗
  27. Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense. arxiv:2303.13408[cs], March 2023. doi: 10.48550/arXiv.2303.13408. URL http://arxiv.org/abs/2303.13408. ↗
  28. Thomas Lavergne, Tanguy Urvoy, and François Yvon. Detecting fake content with relative entropy scoring. PAN, 8:27–31, 2008. find
  29. Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim. Who wrote this code? watermarking for code generation. arXiv preprint arXiv:2305.15060, 2023.
  30. Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097, 2022a.
  31. Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive Decoding: Open-ended Text Generation as Optimization. arxiv:2210.15097[cs], October 2022b. doi: 10.48550/arXiv.2210.15097. URL http://arxiv.org/abs/2210.15097. ↗
  32. Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. GPT detectors are biased against non-native English writers. arxiv:2304.02819[cs], April 2023. doi: 10.48550/arXiv.2304.02819. URL http://arxiv.org/abs/2304.02819. ↗
  33. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
  34. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature. January 2023. doi: 10.48550/arXiv.2301.11305. URL https://arxiv.org/abs/2301.11305v1. ↗
  35. Annie Palmer. People are using A.I. chatbots to write Amazon reviews. CNBC, April 2023. URL https://www.cnbc.com/2023/04/25/amazon-reviews-are-being-written-by-ai-chatbots.html. ↗
  36. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers. In Advances in Neural Information Processing Systems, volume 34, pp. 4816–4828. Curran Associates…
  37. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust Speech Recognition via Large-Scale Weak Supervision. arxiv:2212.04356[cs, eess], December 2022. doi: 10.48550/arXiv.2212.04356. URL http://arxiv.org/abs/2212.04356. ↗
  38. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683 [cs, stat], July 2020. URL http://arxiv.org/abs/1910.10683. ↗
  39. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can AI-Generated Text be Reliably Detected? arxiv:2303.11156[cs], March 2023. doi: 10.48550/arXiv.2303.11156. URL http://arxiv.org/abs/2303.11156. ↗
  40. Bruce Schneier. Using AI to Scale Spear Phishing, August 2021. URL https://www.schneier.com/blog/archives/2021/08/using-ai-to-scale-spear-phishing.html. ↗
  41. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The Curse of Recursion: Training on Generated Data Makes Models Forget. arxiv:2305.17493[cs], May 2023. doi: 10.48550/arXiv.2305.17493. URL http://arxiv.org/abs/2305.17493. ↗
  42. Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A Contrastive Framework for Neural Text Generation. In Advances in Neural Information Processing Systems, October 2022. URL https://openreview.net/forum?id=V88BafmH9Pj. ↗
  43. Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. The Science of Detecting LLM-Generated Texts. arxiv:2303.07205[cs], March 2023. doi: 10.48550/arXiv.2303.07205. URL http://arxiv.org/abs/2303.07205. — [doi:10.48550/arxiv.2303.07205](https://doi.org/10.48550/arxiv.2303.07205) ↗
  44. Edward Tian. Gptzero update v1, January 2023. URL https://gptzero.substack.com/p/gptzero-update-v1. ↗
  45. Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. Label Studio: Data labeling software, 2020-2022. URL https://github.com/heartexlabs/label-studio. Open source software available from https://github.com/heartexlabs/label-studio. ↗
  46. Umut Topkara, Mercan Topkara, and Mikhail J. Atallah. The hiding virtues of ambiguity: Quantifiably resilient watermarking of natural language text through synonym substitutions. In Proceedings of the 8th Workshop on Multimedia and Security, MM&Sec ’06, pp. 164–174, New York, NY, USA, September… — [doi:10.1145/1161366.1161397](https://doi.org/10.1145/1161366.1161397) find
  47. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv prepri…
  48. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  49. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Alexander M. Rush, and Thomas Wolf. The alignment handbook. https://github.com/huggingface/alignment-handbook, 2023. ↗
  50. Lav R. Varshney, Nitish Shirish Keskar, and Richard Socher. Limits of Detecting Text Generated by Large-Scale Language Models. In 2020 Information Theory and Applications Workshop (ITA), pp. 1–5, February 2020. doi: 10.1109/ITA50056.2020.9245012. find
  51. Ashish Venugopal, Jakob Uszkoreit, David Talbot, Franz Och, and Juri Ganitkevitch. Watermarking the Outputs of Structured Prediction with an application in Statistical Machine Translation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 1363–1372, Edinb… find
  52. Vicuna-Team. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna. ↗
  53. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319, 2019.
  54. John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-kirkpatrick. Paraphrastic Representations at Scale. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 379–388, Abu Dhabi, UAE, December 2022. Association for Computational L… find
  55. Max Wolff and Stuart Wolff. Attacking Neural Text Detectors. arxiv:2002.11768[cs], January 2022. doi: 10.48550/arXiv.2002.11768. URL http://arxiv.org/abs/2002.11768. — [doi:10.48550/arxiv.2002.11768](https://doi.org/10.48550/arxiv.2002.11768) ↗
  56. Claire Woodcock. AI Is Tearing Wikipedia Apart, May 2023. URL https://www.vice.com/en/article/v7bdba/ai-is-tearing-wikipedia-apart. ↗
  57. KiYoon Yoo, Wonhyuk Ahn, Jiho Jang, and Nojun Kwak. Robust natural language watermarking through invariant features. arXiv preprint arXiv:2305.01904, 2023. — [arXiv:2305.01904](https://arxiv.org/abs/2305.01904) find
  58. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open Pre-trained Tr… — OPT: Open Pre-trained Transformer Language Models
  59. Zachary Ziegler, Yuntian Deng, and Alexander Rush. Neural Linguistic Steganography. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1210–1215, Hong Kong, China, Nov… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kirchenbauer-2023-reliability-watermarks-large-language-models/