Elamur Research
☰
home / papers / 2023 / Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense

Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2303.13408

Summary

Tools that flag machine-written text include watermarks, statistical detectors such as DetectGPT and GPTZero, and trained classifiers like OpenAI's. They are increasingly used against fake content and plagiarism, but nobody had tested them systematically against rewriting.

The authors built DIPPER, an 11B-parameter discourse-level paraphraser. It was fine-tuned on the PAR3 corpus, in which different English translations of the same foreign novel serve as paragraph-aligned paraphrase pairs. DIPPER can condition on context such as the original prompt, and it offers two inference-time controls: lexical diversity and content reordering.

DIPPER was used to rewrite outputs from three LLMs, including GPT3.5-davinci-003. Every tested detector then missed most of the AI-written text, while automatic metrics and human raters confirmed that the meaning was largely preserved.

As a defense, the paper proposes that API providers store all their generations and compare any suspect text against this archive using BM25 or P-SP. A paraphrase keeps its content, so the search still locates the source. This also challenges the claim that detection must fail once LLM output matches human quality. Models, code and data are released.

Key points

  • On GPT2-XL outputs, DetectGPT's detection rate fell from 70.3% to 4.6% after DIPPER paraphrasing, with the false positive rate held at 1%. The authors argue for fixed low-FPR reporting rather than AUC-ROC.
  • In a retrieval corpus of 15M T5-XXL generations (PG19 and Wikipedia splits of RankGen training data), the method caught 97.3% of PG19 and 80.4% of Wikipedia paraphrases at 1% FPR.
  • Growing the database from 1M to 15M entries cost little accuracy: PG19 went from 98.3 to 97.3 and Wikipedia from 90.0 to 80.4. BM25 consistently beat P-SP.
  • Query length matters. With 20-word queries, BM25 detected under 25% of paraphrases, and performance levels off at around 50 tokens.
  • For ChatGPT-scale use, the authors estimate about 5TB of storage per month and roughly 130 seconds per lookup on a CPU-only MacBook Pro. Stated limitations are privacy risk, no coverage of open-source models like LLAMA, and the cost of maintaining the infrastructure (NeurIPS 2023).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Cited by

170 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 166 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.13408 · arXiv 2303.13408

@misc{krishna2023paraphrasing,
  title = {{Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense}},
  author = {Krishna, Kalpesh and Song, Yixiao and Karpinska, Marzena and Wieting, John and Iyyer, Mohit},
  year = {2023},
  doi = {10.48550/arxiv.2303.13408},
  eprint = {2303.13408},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.13408}
}

References

119 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Abdelnabi and M. Fritz. Adversarial watermarking transformer: Towards tracing text provenance with data hiding. In 2021 IEEE Symposium on Security and Privacy (SP), pages 121–140, 2021. find
  2. E. Agirre, C. Banea, D. Cer, M. Diab, A. Gonzalez-Agirre, R. Mihalcea, G. Rigau, and J. Wiebe. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 2016. doi: 10.18653/v1/S16-1081. URL https://www.aclweb.org/anthology/S16-1081. ↗ our card
  3. R. R. Agrawal, M. Turchi, and M. Negri. Contextual handling in neural machine translation: Look behind, ahead and on both sides. In 21st annual conference of the European association for machine translation, pages 11–20, 2018. find
  4. S. Badaskar, S. Agarwal, and S. Arora. Identifying real or fake articles: Towards better language modeling. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-II, 2008. URL https://aclanthology.org/I08-2115. ↗
  5. A. Bakhtin, S. Gross, M. Ott, Y. Deng, M. Ranzato, and A. Szlam. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351, 2019. ↗
  6. E. Bandel, R. Aharonov, M. Shmueli-Scheuer, I. Shnayderman, N. Slonim, and L. Ein-Dor. Quality controlled paraphrase generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 596–609, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.45. URL https://aclanthology.org/2022.acl-long.45. ↗
  7. E. Bassani. retriv: A user-friendly and efficient search engine in python., 2022. URL https://github.com/AmenRa/retriv. ↗
  8. D. Beresneva. Computer-generated text detection using machine learning: A systematic review. In 21st International Conference on Applications of Natural Language to Information Systems, NLDB, pages 421–426. Springer, 2016. find
  9. R. Bhagat and E. Hovy. Squibs: What is a paraphrase? Computational Linguistics, 39(3):463– 472, 2013. doi: 10.1162/COLI_a_00166. URL https://www.aclweb.org/anthology/J13-3001. ↗ our card
  10. J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax. ↗
  11. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. find
  12. N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650. USENIX Association, Aug. 2021. ISBN 978-1-939133-24-3. URL https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting. ↗ our card
  13. S. Chakraborty, A. S. Bedi, S. Zhu, B. An, D. Manocha, and F. Huang. On the possibilities of ai-generated text detection. arXiv preprint arXiv:2304.04736, 2023. ↗
  14. M. Chen, Q. Tang, S. Wiseman, and K. Gimpel. Controllable paraphrase generation with a syntactic exemplar. In Proceedings of the Association for Computational Linguistics, 2019. doi: 10.18653/v1/P19-1599. URL https://www.aclweb.org/anthology/P19-1599. ↗ our card
  15. M. Connor and D. Roth. Context sensitive paraphrasing with a global unsupervised classifier. In European Conference on Machine Learning, pages 104–115. Springer, 2007. find
  16. P. Damodaran. Parrot: Paraphrase generation for nlu., 2021. URL https://github.com/PrithivirajDamodaran/Parrot_Paraphraser. ↗
  17. Y. Deng, A. Bakhtin, M. Ott, A. Szlam, and M. Ranzato. Residual energy-based models for text generation. In International Conference on Learning Representations, 2020. find
  18. A. Devaraj, I. Marshall, B. Wallace, and J. J. Li. Paragraph-level simplification of medical texts. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4972–4984, 2021. doi: 10.18653/v1/2021.naacl-main.395. URL https://aclanthology.org/2021.naacl-main.395. ↗
  19. T. Dopierre, C. Gravier, and W. Logerais. PROTAUGMENT: Unsupervised diverse short-texts paraphrasing for intent detection meta-learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2454–2466, Online, Aug. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.191. URL https://aclanthology.org/2021.acl-long.191. ↗
  20. A. Fan, Y. Jernite, E. Perez, D. Grangier, J. Weston, and M. Auli. Eli5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, 2019. find our card
  21. S. Garg, S. Prabhu, H. Misra, and G. Srinivasaraghavan. Unsupervised contextual paraphrase generation using lexical control and reinforcement learning. arXiv preprint arXiv:2103.12777, 2021. ↗
  22. S. Gehrmann, H. Strobelt, and A. Rush. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-3019. URL https://aclanthology.org/P19-3019. ↗ our card
  23. D. Gomes, M. Costa, D. Cruz, J. Miranda, and S. Fontes. Creating a billion-scale searchable web archive. In Proceedings of the 22nd International Conference on World Wide Web, pages 1059–1066, 2013. find
  24. Google. How google search organizes information, 2023. URL https://www.google.com/search/howsearchworks/how-search-works/organizing-information. ↗
  25. R. Goutham. High-quality sentence paraphraser using transformers in nlp, 2021. https://github.com/ramsrigouthamg/Questgen.ai/tree/master/NewModels/T5LargeParaphraser. ↗
  26. T. Goyal and G. Durrett. Neural syntactic preordering for controlled paraphrase generation. Proceedings of the Association for Computational Linguistics, 2020. URL https://arxiv.org/abs/2005.02013. ↗ our card
  27. E. Grechnikov, G. Gusev, A. Kustarev, and A. Raigorodsky. Detection of artificial texts. RCDL2009 Proceedings. Petrozavodsk, pages 306–308, 2009. find
  28. A. Grinbaum and L. Adomaitis. The ethical need for watermarks in machine-generated language. arXiv preprint arXiv:2209.03118, 2022. ↗
  29. URL A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. In Proceedings of the International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rygGQyrFvH. ↗
  30. T. Hosking, H. Tang, and M. Lapata. Hierarchical sketch induction for paraphrase generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2489–2501, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.178. URL https://aclanthology.org/2022.acl-long.178. ↗
  31. J. House. Text and context in translation. Journal of pragmatics, 38(3):338–358, 2006. find
  32. D. Hovy. The enemy in your own camp: How well can we detect statistically-generated fake reviews – an adversarial study. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Aug. 2016. doi: 10.18653/v1/P16-2057. URL https://aclanthology.org/P16-2057. ↗
  33. J. E. Hu, R. Rudinger, M. Post, and B. Van Durme. Parabank: Monolingual bitext generation and sentential paraphrasing via lexically-constrained neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6521–6528, 2019. find
  34. K.-H. Huang and K.-W. Chang. Generating syntactically controlled paraphrases without using annotated parallel pairs. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1022–1033, Online, Apr. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.88. URL https://aclanthology.org/2021.eacl-main.88. ↗
  35. D. Ippolito, D. Duckworth, C. Callison-Burch, and D. Eck. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808–1822, July 2020. doi: 10.18653/v1/2020.acl-main.164. URL https://aclanthology.org/2020.acl-main.164. ↗ our card
  36. M. Iyyer, J. Wieting, K. Gimpel, and L. Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1875–1885, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1170. URL https://www.aclweb.org/anthology/N18-1170. ↗ our card
  37. S. Jean, S. Lauly, O. Firat, and K. Cho. Does neural machine translation benefit from larger context? arXiv preprint arXiv:1704.05135, 2017. ↗
  38. S. Jean, A. Bapna, and O. Firat. Fill in the blanks: Imputing missing sentences for larger-context neural machine translation. arXiv preprint arXiv:1910.14075, 2019. ↗
  39. J. Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535–547, 2019. find our card
  40. M. Junczys-Dowmunt. Microsoft translator at WMT 2019: Towards large-scale document-level neural machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 225–233, Florence, Italy, Aug. 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-5321. URL https://aclanthology.org/W19-5321. ↗
  41. V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, 2020. find our card
  42. A. Kazemnejad, M. Salehi, and M. Soleymani Baghshah. Paraphrase generation by learning how to edit from samples. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. acl-main.535. URL https://aclanthology.org/2020.acl-main.535. ↗
  43. O. Khattab and M. Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39–48, 2020. find our card
  44. J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein. A watermark for large language models. arXiv preprint arXiv:2301.10226, 2023a. ↗ our card
  45. J. Kirchenbauer, J. Geiping, Y. Wen, M. Shu, K. Saifullah, K. Kong, K. Fernando, A. Saha, M. Goldblum, and T. Goldstein. On the reliability of watermarks for large language models. arXiv preprint arXiv:2306.04634, 2023b. ↗
  46. R. Koike, M. Kaneko, and N. Okazaki. Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples. arXiv preprint arXiv:2307.11729, 2023. ↗
  47. K. Krishna, J. Wieting, and M. Iyyer. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 737–762, 2020. find our card
  48. K. Krishna, Y. Chang, J. Wieting, and M. Iyyer. RankGen: Improving text generation with large ranking models. arXiv preprint arXiv:2205.09726, 2022a. ↗
  49. K. Krishna, D. Nathani, X. Garcia, B. Samanta, and P. Talukdar. Few-shot controllable style transfer for low-resource multilingual settings. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7439–7468, 2022b. find
  50. S. Kuang, D. Xiong, W. Luo, and G. Zhou. Modeling coherence for neural machine translation with dynamic and topic caches. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, Aug. 2018. URL https://aclanthology.org/C18-1050. ↗
  51. A. Kumar, S. Bhattamishra, M. Bhandari, and P. Talukdar. Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation. In Conference of the North American Chapter of the Association for Computational Linguistics, 2019. doi: 10.18653/v1/N19-1363. URL https://www.aclweb.org/anthology/N19-1363. ↗ our card
  52. A. Kumar, K. Ahuja, R. Vadapalli, and P. Talukdar. Syntax-guided controlled generation of paraphrases. Transactions of the Association for Computational Linguistics, 2020. find our card
  53. T. Kumarage, P. Sheth, R. Moraffah, J. Garland, and H. Liu. How reliable are ai-generated-text detectors? an assessment framework using evasive soft prompts. arXiv preprint arXiv:2310.05095, 2023. ↗
  54. P. Laban, T. Schnabel, P. Bennett, and M. A. Hearst. Keep it simple: Unsupervised simplification of multi-paragraph text. In Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021. acl-long.498. URL https://aclanthology.org/2021.acl-long.498. ↗
  55. V. Lakshman, C. H. Teo, X. Chu, P. Nigam, A. Patni, P. Maknikar, and S. Vishwanathan. Embracing structure in data for billion-scale semantic product search. arXiv preprint arXiv:2110.06125, 2021. ↗
  56. T. Lavergne, T. Urvoy, and F. Yvon. Detecting fake content with relative entropy scoring. PAN, 8: 27–31, 2008. find
  57. M. Lewis, M. Ghazvininejad, G. Ghosh, A. Aghajanyan, S. Wang, and L. Zettlemoyer. Pre-training via paraphrasing. Advances in Neural Information Processing Systems, 33:18470–18481, 2020. find
  58. Z. Li, X. Jiang, L. Shang, and H. Li. Paraphrase generation with deep reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3865–3878, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1421. URL https://aclanthology.org/D18-1421. ↗ our card
  59. Z. Li, X. Jiang, L. Shang, and Q. Liu. Decomposable neural paraphrase generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3403–3414, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1332. URL https://aclanthology.org/P19-1332. ↗
  60. Z. Lin, Y. Cai, and X. Wan. Towards document-level paraphrase generation with sentence rewriting and reordering. In Findings of the Association for Computational Linguistics: EMNLP 2021, Nov. 2021. doi: 10.18653/v1/2021.findings-emnlp.89. URL https://aclanthology.org/2021.findings-emnlp.89. ↗
  61. A. Liu, L. Pan, X. Hu, S. Meng, and L. Wen. A semantic invariant robust watermark for large language models. arXiv preprint arXiv:2310.06356, 2023. ↗
  62. N. Lu, S. Liu, R. He, and K. Tang. Large language models can be guided to evade ai-generated text detection. arXiv preprint arXiv:2305.10847, 2023. ↗
  63. E. Mansimov, G. Melis, and L. Yu. Capturing document context inside sentence-level neural machine translation models with self-training. In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 143–153, Nov. 2021. doi: 10.18653/v1/2021.codi-main.14. URL https://aclanthology.org/2021.codi-main.14. ↗
  64. S. Maruf, F. Saleh, and G. Haffari. A survey on document-level neural machine translation: Methods and evaluation. ACM Computing Surveys (CSUR), 54(2):1–36, 2021. find
  65. A. Max. Sub-sentencial paraphrasing by contextual pivot translation. In Proceedings of the 2009 Workshop on Applied Textual Inference (TextInfer), pages 18–26, 2009. find
  66. Y. Meng, X. Ao, Q. He, X. Sun, Q. Han, F. Wu, C. Fan, and J. Li. ConRPG: Paraphrase generation using contexts as regularizer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2551–2562, Nov. 2021. doi: 10.18653/v1/2021.emnlp-main.199. URL https://aclanthology.org/2021.emnlp-main.199. ↗
  67. H. M. Meral, B. Sankur, A. S. Özsoy, T. Güngör, and E. Sevinç. Natural language watermarking via morphosyntactic alterations. Computer Speech & Language, 23(1):107–125, 2009. find
  68. S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. find our card
  69. L. Miculicich, D. Ram, N. Pappas, and J. Henderson. Document-level neural machine translation with hierarchical attention networks. In Empirical Methods in Natural Language Processing, 2018. doi: 10.18653/v1/D18-1325. URL https://aclanthology.org/D18-1325. ↗ our card
  70. E. Mitchell, Y. Lee, A. Khazatsky, C. D. Manning, and C. Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305, 2023. ↗ our card
  71. S. B. Needleman and C. D. Wunsch. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of molecular biology, 48(3):443–453, 1970. find
  72. NewYorkPost. Chatgpt faces ‘growing pains’ as website traffic drops for first time, 2023. URL https://nypost.com/2023/07/05/chatgpt-website-traffic-drops-for-first-time-in-june. ↗
  73. OpenAI. OpenAI Models - GPT3.5, 2022. URL https://platform.openai.com/docs/models/gpt-3-5. ↗
  74. OpenAI. AI text classifier, ai-text-classifier. Jan 2023a. find
  75. OpenAI. New ways to manage your data in ChatGPT, 2023b. URL https://openai.com/blog/new-ways-to-manage-your-data-in-chatgpt. ↗
  76. URL https://beta.openai.com/L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. ↗
  77. V. Patil, P. Hase, and M. Bansal. Can sensitive information be deleted from llms? objectives for defending against extraction attacks. arXiv preprint arXiv:2309.17410, 2023. ↗
  78. A. Popescu-Belis, S. Loáiciga, C. Hardmeier, and D. Xiong, editors. Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019), Hong Kong, China, Nov. 2019. Association for Computational Linguistics. URL https://aclanthology.org/D19-6500. ↗
  79. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019. URL https://openai.com/blog/better-language-models/. ↗
  80. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67, 2020. find
  81. A. Rajauria. Pegasus fine-tuned for paraphrasing, 2020. URL https://huggingface.co/tuner007/pegasus_paraphrase. ↗
  82. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of Empirical Methods in Natural Language Processing, 2016. doi: 10.18653/v1/D16-1264. URL https://www.aclweb.org/anthology/D16-1264. ↗ our card
  83. Reuters. OpenAI, Google, others pledge to watermark AI content for safety, White House says, 2023a. find
  84. Reuters. EU lawmakers vote for tougher AI rules as draft moves to final stage, 2023b. find
  85. A. Roberts, H. W. Chung, A. Levskaya, G. Mishra, J. Bradbury, D. Andor, S. Narang, B. Lester, C. Gaffney, A. Mohiuddin, et al. Scaling up models and data with T5X and seqio. arXiv preprint arXiv:2203.17189, 2022. ↗
  86. S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, M. Gatford, et al. Okapi at trec-3. Nist Special Publication Sp, 109:109, 1995. find our card
  87. A. Roy and D. Grangier. Unsupervised paraphrasing without translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6033–6039, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1605. URL https://aclanthology.org/P19-1605. ↗
  88. V. S. Sadasivan, A. Kumar, S. Balasubramanian, W. Wang, and S. Feizi. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.11156, 2023. ↗ our card
  89. J. Schulman, B. Zoph, C. Kim, J. Hilton, J. Menick, J. Weng, J. Uribe, L. Fedus, L. Metz, et al. ChatGPT: Optimizing language models for dialogue, 2022. URL https://openai.com/blog/chatgpt. ↗
  90. A. See, A. Pappu, R. Saxena, A. Yerukola, and C. D. Manning. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 843–861, Nov. 2019. doi: 10.18653/v1/K19-1079. URL https://aclanthology.org/K19-1079. ↗
  91. R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. find our card
  92. C. Stokel-Walker. Ai bot chatgpt writes smart essays-should academics worry? Nature, 2022. find
  93. K. Thai, M. Karpinska, K. Krishna, W. Ray, M. Inghilleri, J. Wieting, and M. Iyyer. Exploring document-level literary machine translation with parallel paragraphs from world literature. In Empirical Methods in Natural Language Processing, 2022. find
  94. N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663, 2021. ↗ our card
  95. B. Thompson and M. Post. Paraphrase generation as zero-shot multilingual translation: Disentangling semantic similarity from lexical and syntactic diversity. In Proceedings of the Fifth Conference on Machine Translation, pages 561–570, Online, Nov. 2020. Association for Computational Linguistics. URL https://aclanthology.org/2020.wmt-1.67. ↗ our card
  96. E. Tian. Gptzero: An ai text detector, 2023. URL https://gptzero.me/. ↗
  97. J. Tiedemann and Y. Scherrer. Neural machine translation with extended context. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 82–92, Copenhagen, Denmark, Sept. 2017. doi: 10.18653/v1/W17-4811. URL https://aclanthology.org/W17-4811. ↗ our card
  98. M. Tkachenko, M. Malyuk, A. Holmanyuk, and N. Liubimov. Label Studio: Data labeling software, 2020-2022. URL https://github.com/heartexlabs/label-studio. Open source software available from https://github.com/heartexlabs/label-studio. ↗
  99. M. Topkara, C. M. Taskiran, and E. J. Delp III. Natural language watermarking. In Security, Steganography, and Watermarking of Multimedia Contents VII, volume 5681, pages 441–452, 2005. find
  100. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. ↗
  101. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Proceedings of Advances in Neural Information Processing Systems, pages 5998–6008, 2017. URL http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf. ↗
  102. E. Voita, P. Serdyukov, R. Sennrich, and I. Titov. Context-aware neural machine translation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264–1274, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1117. URL https://aclanthology.org/P18-1117. ↗ our card
  103. E. Voita, R. Sennrich, and I. Titov. Context-aware monolingual repair for neural machine translation. In Empirical Methods in Natural Language Processing, pages 877–886, 2019a. doi: 10.18653/v1/D19-1081. URL https://aclanthology.org/D19-1081. ↗
  104. E. Voita, R. Sennrich, and I. Titov. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proceedings of the Association for Computational Linguistics, 2019b. doi: 10.18653/v1/P19-1116. URL https://aclanthology.org/P19-1116. ↗ our card
  105. L. Wang, Z. Tu, A. Way, and Q. Liu. Exploiting cross-sentence context for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2826–2831, 2017. doi: 10.18653/v1/D17-1301. URL https://aclanthology.org/D17-1301. ↗
  106. J. Wieting and K. Gimpel. ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1042. URL https://www.aclweb.org/anthology/P18-1042. ↗ our card
  107. J. Wieting, T. Berg-Kirkpatrick, K. Gimpel, and G. Neubig. Beyond BLEU:training neural machine translation with semantic similarity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4344–4355, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1427. URL https://aclanthology.org/P19-1427. ↗ our card
  108. J. Wieting, K. Gimpel, G. Neubig, and T. Berg-kirkpatrick. Paraphrastic representations at scale. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 379–388, Abu Dhabi, UAE, Dec. 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.emnlp-demos.38. ↗
  109. S. Witteveen and M. Andrews. Paraphrasing with large language models. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 215–220, Hong Kong, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-5623. URL https://aclanthology.org/D19-5623. ↗
  110. X. Xie, X. Lu, and B. Chen. Multi-task learning for paraphrase generation with keyword and part-of-speech reconstruction. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1234–1243, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.97. URL https://aclanthology.org/2022. findings-acl.97. ↗
  111. H. Xiong, Z. He, H. Wu, and H. Wang. Modeling coherence for discourse neural machine translation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7338–7345, 2019. find
  112. W. Xu, C. Callison-Burch, and C. Napoles. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297, 2015. find our card
  113. K. Yang, D. Liu, W. Lei, B. Yang, H. Zhang, X. Zhao, W. Yao, and B. Chen. Gcpg: A general framework for controllable paraphrase generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 4035–4047, 2022. find
  114. K. Yin, P. Fernandes, A. F. Martins, and G. Neubig. When does translation require context? a data-driven, multilingual exploration. arXiv preprint arXiv:2109.07446, 2021. ↗
  115. K. Yoo, W. Ahn, and N. Kwak. Advancing beyond identification: Multi-bit watermark for language models. arXiv preprint arXiv:2308.00221, 2023. ↗
  116. R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, and Y. Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019. find our card
  117. J. Zhang, H. Luan, M. Sun, F. Zhai, J. Xu, M. Zhang, and Y. Liu. Improving the transformer translation model with document-level context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. doi: 10.18653/v1/D18-1049. URL https://aclanthology.org/D18-1049. ↗ our card
  118. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. ↗
  119. X. Zhao, P. Ananth, L. Li, and Y.-X. Wang. Provable robust watermarking for ai-generated text. arXiv preprint arXiv:2306.17439, 2023. ↗