Elamur Research⌕
☰
home / papers / 2021 / TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation

TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation

Paper · 2021 · Adaku Uchendu, Zeyu Ma, Thai Le et al.

Summary

As neural language models produce increasingly fluent text, telling machine output apart from human writing has become a practical concern, for example in countering fake news and propaganda. Before this work, no shared benchmark existed for studying the problem systematically. The authors present TURINGBENCH, which has three parts: a dataset, two tasks, and a public website with leaderboards.

To build the dataset, they collected 10K human-written news articles, mostly political, from CNN, Washington Post and Kaggle. Each article's title served as a prompt for 19 generators built on 10 architectures, including GPT-1/2/3, GROVER, CTRL, XLM, XLNET, FAIR, TRANSFORMER_XL and PPLM. The two tasks are:

  • Turing Test (TT): binary human-vs-machine classification.
  • Authorship Attribution (AA): identifying which of 20 authors wrote a text.

Five state-of-the-art TT detectors were tested. FAIR_wmt20 and GPT-3 produced the most human-like text, earning the lowest detection F1. Classical and neural AA models also could not fully separate the authors' styles. The benchmark gives the community a common testbed for building stronger, more explainable detectors.

Key points

  • Dataset: 200K samples with 20 labels (Human plus 19 generators). Human source articles were filtered to 200–500 words, leaving a 130K pool, of which 10K were used as prompts.
  • TT task: split into 19 separate human-vs-generator subtasks. AA task: a single 20-way classification.
  • Hardest to detect: FAIR_wmt20 was the most difficult. GPT-3 and the GPT-2 variants (small, medium, large, XL) also scored below 0.62 F1.
  • Detector behavior: GPT-2 detector and GLTR fell below 0.4 F1 on some subtasks and often labeled sophisticated machine text as human. Text length (100–400 words) showed no clear link to performance, except for RoBERTa.
  • Style analysis: a PCA of LIWC psycholinguistic features shows the 20 authors overlapping heavily. Human texts spread widely across the plot, while each generator stays in a narrow region.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓Pennsylvania State UniversityState College · United States22 works · 4,878 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Evaluation & benchmarks · 685 works Text generation · 145 works

Cite

Cite as Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang and Dongwon Lee. 2021. TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation. Findings of the Association for Computational Linguistics: EMNLP 2021.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-emnlp.172

@inproceedings{uchendu2021turingbench,
  title = {{TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation}},
  author = {Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang and Dongwon Lee},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.172},
  url = {https://doi.org/10.18653/v1/2021.findings-emnlp.172}
}

References

73 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Haotian Mai, Fuming Fang, Huy H Nguyen, Junichi Yamagishi, and Isao Echizen. 2020. Generating sentiment-preserving fake online reviews using neural language models and their human-and machine-based detection. In International Conference on Advanced Information Networking and… find
  2. Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2018. code2seq: Generating sequences from structured representations of code. In International Conference on Learning Representations. find
  3. Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and Arthur Szlam. 2019. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351. — [arXiv:1906.03351](https://arxiv.org/abs/1906.03351) find
  4. Meghana Moorthy Bhat and Srinivasan Parthasarathy. 2020. How effectively can machines defend against machine-generated fake news? an empirical study. In Proceedings of the First Workshop on Insights from Negative Results in NLP, pages 48–53. find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… our card
  6. Erion Çano and Ondřej Bojar. 2020. Human or machine: Automating human likeliness evaluation of nlg texts. arXiv preprint arXiv:2006.03189. — [arXiv:2006.03189](https://arxiv.org/abs/2006.03189) find
  7. Peng-Jen Chen, Ann Lee, Changhan Wang, Naman Goyal, Angela Fan, Mary Williamson, and Jiatao Gu. 2020. Facebook ai’s wmt20 news translation task submission. In Proceedings of the Fifth Conference on Machine Translation, pages 113–125. find
  8. Rosa María Coyotl-Morales, Luis Villaseñor-Pineda, Manuel Montes-y Gómez, and Paolo Rosso. 2006. Authorship attribution using word sequences. In Iberoamerican Congress on Pattern Recognition, pages 844–853. Springer. find
  9. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988. find
  10. Avisha Das and Rakesh Verma. 2018. Automated email generation for targeted attacks using natural language. In TA-COS 2018: 2nd Workshop on Text Analytics for Cybersecurity and Online Safety, page 23. find
  11. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations. our card
  12. Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc’Aurelio Ranzato. 2019. Residual energy-based models for text generation. In International Conference on Learning Representations. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  14. H. Jair Escalante, Thamar Solorio, and Manuel Montes y Gómez. 2011. Local histograms of character ngrams for authorship attribution. In Proceedings of the 49th Annual Meeting for the Association for Computational Linguistics: Human Language Technologies, pages 288–298, Portland, Oregon. ACL. find
  15. Maël Fabien, Esa ú Villatoro-Tello, Petr Motlicek, and Shantipriya Parida. 2020. Bertaa: Bert fine-tuning for authorship attribution. In Proceedings of the 17th International Conference on Natural Language Processing, CONF. ACL. find
  16. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898. find
  17. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961. find
  18. Elisa Ferracane, Su Wang, and Raymond Mooney. 2017. Leveraging discourse information effectively for authorship attribution. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 584–593. find
  19. Tommaso Fornaciari and Massimo Poesio. 2014. Identifying fake amazon reviews as learning from crowds. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 279–287. find
  20. Cristina Garbacea and Qiaozhu Mei. 2020. Neural language generation: Formulation, methods, and evaluation. arXiv preprint arXiv:2007.15780. — [arXiv:2007.15780](https://arxiv.org/abs/2007.15780) find
  21. Zhenhao Ge, Yufang Sun, and Mark Smith. 2016. Authorship attribution using a neural network language model. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  22. Maarten Lambers and Cor J Veenman. 2009. Forensic authorship attribution using compression distances to prototypes. In International Workshop on Computational Forensics, pages 13–24. Springer. find
  23. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019. Gltr: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116. our card
  24. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  25. Jade Goldstein, Ransom Winder, and Roberta Sabin. 2009. Person identification from text and speech genre samples. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 336–344. find
  26. Oren Halvani, Lukas Graner, Roey Regev, and Philipp Marquardt. 2020. An improved topic masking technique for authorship analysis. arXiv preprint arXiv:2005.06605. — [arXiv:2005.06605](https://arxiv.org/abs/2005.06605) find
  27. Julian Hitschler, Esther Van Den Berg, and Ines Rehbein. 2017. Authorship attribution with convolutional neural networks and pos-eliding. In Proceedings of the Workshop on Stylistic Variation, pages 53–58. find
  28. John Houvardas and Efstathios Stamatatos. 2006. Ngram feature selection for authorship identification. In International conference on artificial intelligence: Methodology, systems, and applications, pages 77– 86. Springer. find
  29. Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. our card
  30. Fereshteh Jafariakinabad and Kien A Hua. 2019. Style-aware neural model with application in authorship attribution. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pages 325–328. IEEE. find
  31. Fereshteh Jafariakinabad and Kien A Hua. 2020. A self-supervised representation learning of sentence structure for authorship attribution. arXiv preprint arXiv:2010.06786. — [arXiv:2010.06786](https://arxiv.org/abs/2010.06786) find
  32. Fereshteh Jafariakinabad, Sansiri Tarnpradab, and Kien A Hua. 2019. Syntactic recurrent neural network for authorship attribution. arXiv preprint arXiv:1902.09723. — [arXiv:1902.09723](https://arxiv.org/abs/1902.09723) find
  33. Ganesh Jawahar, Muhammad Abdul-Mageed, and VS Laks Lakshmanan. 2020. Automatic detection of machine generated text: A critical survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2296–2309. our card
  34. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858. our card
  35. Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203–1213. our card
  36. Jiwei Li, Myle Ott, Claire Cardie, and Eduard Hovy. 2014. Towards a general rule for identifying deceptive opinion spam. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1566– 1576. find
  37. Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew Mc-Dermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. 2019a. Clinically accurate chest x-ray report generation. In Machine Learning for Healthcare Conference, pages 249–269. PMLR. find
  38. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  39. A Pastor López-Monroy, Fabio A González, and Thamar Solorio. 2020. Early author profiling on twitter using profile features with multi-resolution. Expert Systems with Applications, 140:112909. find
  40. Asad Mahmood, Faizan Ahmad, Zubair Shafiq, Padmini Srinivasan, and Fareed Zaffar. 2019. A girl has no name: Automated authorship obfuscation using mutant-x. Proceedings on Privacy Enhancing Technologies, 2019(4):54–71. find
  41. Enrique Manjavacas, Jeroen De Gussem, Walter Daelemans, and Mike Kestemont. 2017. Assessing the stylistic properties of neurally generated text in authorship attribution. arXiv preprint arXiv:1708.05536. find
  42. Ahmadreza Mosallanezhad, Kai Shu, and Huan Liu. 2020. Topic-preserving synthetic news generation: An adversarial deep reinforcement learning approach. arXiv preprint arXiv:2010.16324. — [arXiv:2010.16324](https://arxiv.org/abs/2010.16324) find
  43. Shaoor Munir, Brishna Batool, Zubair Shafiq, Padmini Srinivasan, and Fareed Zaffar. 2021. Through the looking glass: Learning to attribute synthetic text generated by language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main… find
  44. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair’s wmt19 news translation task submission. arXiv preprint arXiv:1907.06616. our card
  45. Kai Shu, Yichuan Li, Kaize Ding, and Huan Liu. 2021. Fact-enhanced synthetic news generation. In AAAI. find
  46. Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social media: A data mining perspective. ACM SIGKDD explorations newsletter, 19(1):22–36. find
  47. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL Irene Solaiman, Miles Brundage, Jack Clark, Amanda https://s3-us-west-2.amazonaws. com/openai-Askell, Ariel Herbert-Voss, Jeff Wu, Alec Rad-assets/researchcover… ↗
  48. Kreps, et al. 2019. Release strategies and the social impacts of language models. arXiv preprint Alec Radford, Jeffrey Wu, Rewon Child, David Luan, arXiv:1908.09203. our card
  49. Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  50. OpenAI Efstathios Stamatatos. 2009a. Intrinsic plagiarism de-blog, 1(8):9. tection using character n-gram profiles. In 3rd PAN WORKSHOP. find
  51. UNCOVERING PLAGIARISM, AU-Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and THORSHIP AND SOCIAL SOFTWARE MISUSE. Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of Efstathios Stamatatos. 2009b. A survey of modern au-the 2016 Conference on Empirical… our card
  52. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying Alan M Turing. 2009. Computing machinery and in-shades: Analyzing language in fake news and polit-telligence. In Parsing the turing test, pages 23–65. ical fact-checking. In Proceedings of the 2017 co… find
  53. Adaku Uchendu, Jeffery Cao, Qiaozhi Wang, Bo Luo, and Dongwon Lee. 2019. Characterizing man-made vs. machine-made chatbot dialogs. In Proceedings of the Int’l Conf. on Truth and Trust Online (TTO). find
  54. Upendra Sapkota, Steven Bethard, Manuel Montes y Gomez, and Thamar Solorio. 2015. Not all character n-grams are created equal: A study in authorship attribution. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn… find
  55. Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. 2020. Authorship attribution for neural text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). our card
  56. Upendra Sapkota, Thamar Solorio, Manuel Montes y Gomez, and Steven Bethard. 2016. Domain adaptation for authorship attribution: Improved structural correspondence learning. In Association for Computational Linguistics (ACL), Berlin, Germany. ACL. find
  57. Onur Varol, Emilio Ferrara, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2017. Online human-bot interactions: Detection, estimation, and characterization. In Proceedings of the International AAAI Conference on Web and Social Media. find
  58. Tal Schuster, Roei Schuster, Darsh J Shah, and Regina Barzilay. 2020. The limitations of stylometry for detecting machine-generated fake news. Computational Linguistics, 46(2):499–510. find
  59. Lav R Varshney, Nitish Shirish Keskar, and Richard Socher. 2020. Limits of detecting text generated by large-scale language models. In 2020 Information Theory and Applications Workshop (ITA), pages 1– 5. IEEE. find
  60. Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 843–861. find
  61. Prasha Shrestha, Sebastian Sierra, Fabio Gonzalez, Manuel Montes, Paolo Rosso, and Thamar Solorio. 2017. Convolutional neural networks for authorship attribution of short texts. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2,… find
  62. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30:5998–6008. our card
  63. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355. our card
  64. William Yang Wang. 2017. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426. our card
  65. Joseph Weizenbaum. 1966. Eliza—a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1):36–45. find
  66. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, pages arXiv–1910. our card
  67. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754–5764. our card
  68. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems, pages 9051–9062. our card
  69. Chunxia Zhang, Xindong Wu, Zhendong Niu, and Wei Ding. 2014. Authorship identification from unstructured texts. Knowledge-Based Systems, 66:99–111. find
  70. Richong Zhang, Zhiyuan Hu, Hongyu Guo, and Yongyi Mao. 2018. Syntax encoding with application in authorship attribution. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2742–2753. find
  71. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. 2020. Dialogpt: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational… our card
  72. Wanjun Zhong, Duyu Tang, Zenan Xu, Ruize Wang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020. Neural deepfake detection with factual structure of text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  73. Zhixuan Zhou, Huankang Guan, Meghana Moorthy Bhat, and Justin Hsu. 2019. Fake news detection via nlp is vulnerable to adversarial attacks. arXiv preprint arXiv:1901.09657. — [arXiv:1901.09657](https://arxiv.org/abs/1901.09657) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/uchendu-2021-turingbench-benchmark-environment-turing-test/