Elamur Research⌕
☰
home / papers / 2020 / Authorship Attribution for Neural Text Generation

Authorship Attribution for Neural Text Generation

Summary

This EMNLP 2020 study treats the detection of neural text generators as an authorship problem. It asks three questions. First, do two texts come from the same source? Second, was a text produced by a person or by a machine? Third, which of k = 8 candidate generators produced a given text? The authors assembled a political-news corpus with nine sources. The human part is professionally written articles. The other eight parts were produced by CTRL, GPT, GPT2, GROVER, XLM, XLNET, PPLM and FAIR, each prompted with the human headlines and asked for 500 words. The first two questions were framed as binary classification and the third as multi-class classification. The authors tested classical learners on stylistic features, several CNN/RNN architectures, n-gram and POS-based attribution methods, and pretrained detectors (RoBERTa, GROVER-DETECT). Most generators turned out to be easy to separate from human writing. GPT2, GROVER and FAIR produced more convincing text and confused the classifiers most. The work matters for tracing disinformation back to a specific generator, and the code and data are publicly released.

Key points

  • The human data consists mostly of political news: 819 CNN, 132 Washington Post and 113 New York Times articles. Their headlines served as prompts for all eight generators.
  • Same-source detection (P1) is easier when the classes are balanced, reaching F1 = 0.9813. The imbalanced setting is noticeably harder.
  • For human vs. machine detection (P2), fine-tuned RoBERTa performed best overall and reached F1 = 0.9792 on GPT2 text. GROVER-DETECT did well only on GROVER output (F1 = 0.8135).
  • For attributing a text to a specific generator (P3), Random Forest on linguistic features achieved the highest average F1 (0.9042). CTRL, XLM and XLNET were easy to identify, while Human, GROVER, FAIR and GPT2 were the hardest.
  • The most useful stylistic cues were word count, articles, periods, words per sentence, auxiliary verbs, prepositions and commas. A topic-based baseline (LDA topics with bigrams and trigrams) scored only 0.38 F1. This suggests topic differences alone do not explain the attribution results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

77 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 71 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.673

@inproceedings{uchendu2020authorship,
  title = {{Authorship Attribution for Neural Text Generation}},
  author = {Uchendu, Adaku and Le, Thai and Shu, Kai and Lee, Dongwon},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.673},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.673}
}

References

54 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hanan Alshaher and Jinsheng Xu. 2020. A new term weight scheme and ensemble technique for authorship identification. In Proceedings of the 2020 the 4th International Conference on Compute and Data Analysis, pages 123–130. find
  2. Bander Alsulami, Edwin Dauber, Richard Harang, Spiros Mancoridis, and Rachel Greenstadt. 2017. Source code authorship attribution using long short-term memory based networks. In European Symposium on Research in Computer Security, pages 65– 82. Springer. find
  3. Grzegorz Baron. 2014. Influence of data discretization on efficiency of bayesian classifier for authorship attribution. Procedia Computer Science, 35:1112– 1121. find
  4. Dainis Boumber, Yifan Zhang, and Arjun Mukherjee. 2018. Experiments with convolutional neural networks for multi-label authorship attribution. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  5. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. find
  6. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: a simple approach to controlled text generation. arXiv preprint arXiv:1912.02164. find our card
  7. Elisa Ferracane, Su Wang, and Raymond Mooney. 2017. Leveraging discourse information effectively for authorship attribution. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 584–593. find
  8. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043. find our card
  9. Oren Halvani, Lukas Graner, Roey Regev, and Philipp Marquardt. 2020. An improved topic masking technique for authorship analysis. arXiv preprint arXiv:2005.06605. — [arXiv:2005.06605](https://arxiv.org/abs/2005.06605) find
  10. Julian Hitschler, Esther van den Berg, and Ines Rehbein. 2017. Authorship attribution with convolutional neural networks and pos-eliding. In Proceedings of the Workshop on Stylistic Variation, pages 53–58. find
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  12. Armin Hoenen and Niko Schenk. 2018. Knowing the author by the company his words keep. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  13. A. Pastor López-Monroy, Fabio A Gonzalez, and Thamar Solorio. 2020. Early author profiling on twitter using profile features with multi-resolution. Expert Systems with Applications, 140:112909. find
  14. Renkui Hou and Chu-Ren Huang. 2017. Stylometric studies based on tone and word length motifs. In Proceedings of the 31st Pacific Asia Conference on Language, Information and Computation, pages 56– 63. find
  15. Enrique Manjavacas, Jeroen De Gussem, Walter Daelemans, and Mike Kestemont. 2017. Assessing the stylistic properties of neurally generated text in authorship attribution. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), pages 116–125. find
  16. Fatma Howedi and Masnizah Mohd. 2014. Text classification for authorship attribution using naive bayes classifier with limited training data. Computer Engineering and Intelligent Systems, 5(4):48–56. find
  17. Fereshteh Jafariakinabad, Sansiri Tarnpradab, and Kien A Hua. 2019. Syntactic recurrent neural network for authorship attribution. arXiv preprint arXiv:1902.09723. — [arXiv:1902.09723](https://arxiv.org/abs/1902.09723) find
  18. Patrick Juola. 2012. Detecting stylistic deception. In Proceedings of the Workshop on Computational Approaches to Deception Detection, pages 91–96. Association for Computational Linguistics. find
  19. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858. find our card
  20. Mike Kestemont. 2014. Function words in authorship attribution. from black magic to theory? In Proceedings of the 3rd Workshop on Computational Linguistics for Literature (CLFL), pages 59–66. find
  21. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. find our card
  22. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  23. Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille. find
  24. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  25. Jiwei Li, Myle Ott, Claire Cardie, and Eduard Hovy. 2014. Towards a general rule for identifying deceptive opinion spam. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1566– 1576. find
  26. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair’s wmt19 news translation task submission. arXiv preprint arXiv:1907.06616. find our card
  27. James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates, 71(2001):2001. find
  28. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  29. Thomas Proisl, Stefan Evert, Fotis Jannidis, Christof Schöch, Leonard Konle, and Steffen Pielström. 2018. Delta vs. n-gram tracing: Evaluating the robustness of authorship attribution methods. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  30. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf. ↗
  31. Yafeng Ren and Donghong Ji. 2017. Neural networks for deceptive opinion spam detection: An empirical study. Information Sciences, 385:213–224. find
  32. Javier Sánchez-Junquera, Luis Villaseñor-Pineda, Manuel Montes-y Gómez, Paolo Rosso, and Efstathios Stamatatos. 2020. Masking domain-specific information for cross-domain deception detection. Pattern Recognition Letters, 135:122–130. find
  33. Yunita Sari, Mark Stevenson, and Andreas Vlachos. 2018. Topic or style? exploring the most useful features for authorship attribution. In Proceedings of the 27th International Conference on Computational Linguistics, pages 343–353. find
  34. Yunita Sari, Andreas Vlachos, and Mark Stevenson. 2017. Continuous n-gram representations for authorship attribution. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 267–273. find
  35. Thomas Schürmann and Peter Grassberger. 1996. Entropy estimation of symbol sequences. Chaos: An Interdisciplinary Journal of Nonlinear Science, 6(3):414–427. find
  36. Yanir Seroussi, Fabian Bohnert, and Ingrid Zukerman. 2012. Authorship attribution with author-aware topic models. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2, pages 264–269. Association for Computational Linguistics. find
  37. Yanir Seroussi, Ingrid Zukerman, and Fabian Bohnert. 2011. Authorship attribution with latent dirichlet allocation. In Proceedings of the fifteenth conference on computational natural language learning, pages 181–189. Association for Computational Linguistics. find
  38. Yanir Seroussi, Ingrid Zukerman, and Fabian Bohnert. 2014. Authorship attribution with topic models. Computational Linguistics, 40(2):269–310. find
  39. Jialin Shao, Adaku Uchendu, and Dongwon Lee. 2019. A reverse turing test for detecting machine-made texts. In Proceedings of the 10th ACM Conference on Web Science, pages 275–279. find
  40. Abhay Sharma, Ananya Nandan, and Reetika Ral-han. 2018. An investigation of supervised learning methods for authorship attribution in short hinglish texts using char & word n-grams. arXiv preprint arXiv:1812.10281. — [arXiv:1812.10281](https://arxiv.org/abs/1812.10281) find
  41. Prasha Shrestha, Sebastian Sierra, Fabio A González, Manuel Montes, Paolo Rosso, and Thamar Solorio. 2017. Convolutional neural networks for authorship attribution of short texts. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume… find
  42. Lucy Simko, Luke Zettlemoyer, and Tadayoshi Kohno. 2018. Recognizing and imitating programmer style: Adversaries in program authorship attribution. Proceedings on Privacy Enhancing Technologies, 2018(1):127–144. find
  43. Kalaivani Sundararajan and Damon Woodard. 2018. What represents “style” in authorship attribution? In Proceedings of the 27th International Conference on Computational Linguistics, pages 2814–2822. find
  44. Michael Tschuggnall and Günther Specht. 2014. Enhancing authorship attribution by utilizing syntax tree profiles. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 195–199. find
  45. Adaku Uchendu, Jeffery Cao, Qiaozhi Wang, Bo Luo, and Dongwon Lee. 2019. Characterizing man-made vs. machine-made chatbot dialogs. In Proceedings of the Int’l Conf. on Truth and Trust Online (TTO). find
  46. Andreas Van Cranenburgh. 2012. Literary authorship attribution with phrase-structure fragments. In Proceedings of the NAACL-HLT 2012 Workshop on Computational Linguistics for Literature, pages 59– 63. find
  47. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754–5764. find our card
  48. Anelka Zečević. 2011. N-gram based text classification according to authorship. In Proceedings of the Second Student Research Workshop associated with RANLP 2011, pages 145–149. find
  49. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems, pages 9051–9062. find our card
  50. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649–657. find our card
  51. Rong Zheng, Jiexun Li, Hsinchun Chen, and Zan Huang. 2006. A framework for authorship identification of online messages: Writing-style features and classification techniques. Journal of the American society for information science and technology, 57(3):378–393. find
  52. Thamar Solorio, Sangita Pillay, Sindhu Raghavan, and Manuel Montes. 2011. Modality specific meta features for authorship attribution in web forum posts. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 156–164. find
  53. Efstathios Stamatatos. 2017. Authorship attribution using text distortion. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1138–1149. find
  54. Balanced P1 P2 P3 \\\#train 68,896 2,985 6,825 \\\#valid 7,656 426 881 \\\#test 19,139 853 1,888 Imbalanced P1 P2 \\\#train 62,157 6,825 \\\#valid 6,907 881 \\\#test 17,266 1,888 *Table 8: Details on Train, Validation and Test Set Splits* find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/uchendu-2020-authorship-attribution-neural-text-generation/