Elamur Research⌕
☰
home / papers / 2020 / The Limitations of Stylometry for Detecting Machine-Generated Fake News

The Limitations of Stylometry for Detecting Machine-Generated Fake News

Paper · 2020 · Tal Schuster, Roei Schuster, Darsh J. Shah et al.

Summary

The paper investigates whether stylometry, meaning detection based on stylistic cues, is able to tell apart harmful and acceptable applications of language models (LMs). Earlier work showed that stylometry can spot machine-written text, but provenance alone says nothing about truthfulness once LMs are routinely used for auto-completion and editing. The authors build two veracity-labeled benchmarks of news articles in which LMs add or alter content.

  • QA-based extension: an LM extends articles with short claims. A simulated malicious user keeps only false claims, and a simulated honest user keeps only correct ones.
  • Statement inversion: a relatively weak LM picks plausible spots where true statements are flipped into false ones.

They test a fine-tuned Grover-Mega detector on both benchmarks. Stylometry reliably identifies machine-written content, even a single generated sentence, but it largely cannot tell whether that content is true. The reason is that LMs, unlike human liars, keep the same style whatever their intent. Humans were also misled, though access to external sources helped. The work argues for fact-checking and metadata-based defenses and for evaluation sets organized around veracity rather than authorship.

Key points

  • The detector is Grover-Mega, a linear classifier on the final-token hidden state of a Transformer LM trained on news. In the zero-shot setting it reaches 0.90 F1 on fully generated GPT-2 XL texts versus WebText articles.
  • Zero-shot, it mostly misses partially generated New York Times articles extended by GPT-2 Medium. F1 drops to 0.45 at 20% machine text and 0.28 at 1%.
  • After adaptive fine-tuning, it achieves 0.94–0.95 F1 even when only one sentence (g = 1%) is machine-generated. In the QA setting, where human answers come from NewsQA, it reaches 0.83 F1.
  • When both the true and the false content are LM-produced, stylometry fails to separate them. A benchmark that assumes LMs are never used legitimately therefore overstates real-world performance.
  • The data are publicly released. The authors recommend broader veracity-oriented benchmarks and stronger non-stylometry methods.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States211 works · 69,000 citations🎓Cornell UniversityIthaca · United States120 works · 59,940 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States69 works · 30,953 citations🎓Tel Aviv UniversityTel Aviv · Israel82 works · 10,655 citations🎓Cornell TechNew York · United States7 works · 1,320 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

Cite

Cite as Tal Schuster, Roei Schuster, Darsh J. Shah and Regina Barzilay. 2020. The Limitations of Stylometry for Detecting Machine-Generated Fake News. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00380

@article{schuster2020limitations,
  title = {{The Limitations of Stylometry for Detecting Machine-Generated Fake News}},
  author = {Tal Schuster, Roei Schuster, Darsh J. Shah and Regina Barzilay},
  journal = {Computational Linguistics},
  year = {2020},
  doi = {10.1162/coli\_a\_00380},
  url = {https://doi.org/10.1162/coli_a_00380}
}

References

54 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. We thank the anonymous reviewers and the members of the MIT NLP group for their helpful comments. R.S. is a member of the Check Point Institute of Information Technology. This work is supported in part by Google’s TensorFlow Research Cloud program; DSO grant DSOCL18002; by the Blavatnik Interdiscipl… find
  2. Abramov, Polina Shafran and Roman V. Yampolskiy. 2019. In Automatic IQ estimation using stylometric methods. In Handbook of Research on Learning in the Age of Transhumanism. IGI Global, pages 32–45, Hershey, PA. find
  3. Afroz, Sadia, Michael Brennan, and Rachel Greenstadt. 2012. Detecting hoaxes, frauds, and deception in writing style online. In Proceedings of 2012 IEEE Symposium on Security and Privacy. pages 461–475, San Francisco, CA, USA. find
  4. Afroz, Sadia, Aylin Caliskan, Rachel Greenstadt, and Dan Mcoy. 2014. Doppelgänger finder: Taking stylometry to the underground. In Proceedings of 2014 IEEE Symposium on Security and Privacy, pages 212–226, IEEE Computer Society, San Jose, CA, USA. find
  5. Augenstein, Isabelle, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims. In Proceedings of the 2019 Conference on Empirical Methods in Natural Langu…
  6. Bakhtin, Anton, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and Arthur Szlam. 2019. Real or fake? Learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351. — [arXiv:1906.03351](https://arxiv.org/abs/1906.03351) find
  7. Benjakob, Omer. 2019. The fake Nazi death camp: Wikipedia’s longest hoax, exposed. https://tinyurl.com/y3m2wd9k. ↗
  8. Haaretz. Bond, Gary D. and Adrienne Y. Lee. 2005. Language of lies in prison: Linguistic classification of prisoners’ truthful and deceptive natural language. Applied Cognitive Psychology, 19(3):313–329. find
  9. Brennan, Michael, Sadia Afroz, and Rachel Greenstadt. 2012. Adversarial stylometry: Circumventing authorship recognition to preserve privacy and anonymity. ACM Transactions on Information and System Security (TISSEC), 15(3):12. find
  10. Caliskan-Islam, Aylin, Richard Harang, Andrew Liu, Arvind Narayanan, Clare Voss, Fabian Yamaguchi, and Rachel Greenstadt. 2015. De-anonymizing programmers via code stylometry. In 24th USENIX Security Symposium (USENIX Security 15), pages 255–270, Washington, D.C. find
  11. Castillo, Carlos, Marcelo Mendoza, and Barbara Poblete. 2011. Information credibility on Twitter. In Proceedings of the 20th International Conference on World Wide Web, pages 67–684. find
  12. Choshen, Leshem, Dan Eldad, Daniel Hershcovich, Elior Sulem, and Omri Abend. 2019. The language of legal and illegal activity on the Darknet. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4271–4279, Florence. von Davier, Matthias. 2019. Training op… find
  13. Dostoyevsky, Fyodor. 1866. find
  14. Crime and Punishment. The Russian Messenger. Enos, Frank, Elizabeth Shriberg, Martin Graciarena, Julia Hirschberg, and Andreas Stolcke. 2007. Detecting deception using critical segments. In International Speech Communication Association, pages 2281–2284, Antwerp, Belgium. find
  15. Feng, Song, Ritwik Banerjee, and Yejin Choi. 2012. Syntactic stylometry for deception detection. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2, ACL ’12, pages 171–175. find
  16. Frank, Mark G., Melissa A. Menasco, and Maureen O’Sullivan. 2008. Human behavior and deception detection. In John G. Voeller, editor, Wiley Handbook of Science and Technology for Homeland Security. Wiley & Sons, Inc., pages 1–13. find
  17. Gehrmann, Sebastian, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Conference of the Association for Computational Linguistics: System Demonstrations, pages 111–116, Florence.
  18. Goswami, Sumit, Sudeshna Sarkar, and Mayur Rustagi. 2009. Stylometric analysis of bloggers’ age and gender. In International AAAI Conference on Web and Social Media, San Jose, CA. find
  19. Gupta, Aditi, Ponnurangam Kumaraguru, Carlos Castillo, and Patrick Meier. 2014. TweetCred: Real-time credibility assessment of content on Twitter. In International Conference on Social Informatics, pages 228–243, Cham. find
  20. Hanselowski, Andreas, P. V. S. Avinesh, Benjamin Schiller, Felix Caspelherr, Debanjan Chaudhuri, Christian M. Meyer, and Iryna Gurevych. 2018. A retrospective analysis of the fake news challenge stance-detection task. In Proceedings of the 27th International Conference on Computational Linguistics,…
  21. House, Patrick. 2019. I, language robot. https://lareviewofbooks.org/article/ i-language-robot/. Los Angeles Review of Books. ↗
  22. Kochkina, Elena, Maria Liakata, and Arkaitz Zubiaga. 2018. All-in-one: Multi-task learning for rumour verification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3402–3413, Santa Fe, NM. find
  23. Liu, Yang and Yi-Fang Wu. 2018. Early detection of fake news on social media through propagation path classification with recurrent and convolutional networks. In AAAI Conference on Artificial Intelligence, pages 354–361, New Orleans, LA. find
  24. Matsumoto, David,, Hyi Sung Hwang, Lisa Skinner, and Mark Frank. 2011. Evaluating truthfulness and detecting deception. FBI Law Enforcement Bulletin, (80):1. find
  25. Mazar, Nina, On Amir, and Dan Ariely. 2008. The dishonesty of honest people: A theory of self-concept maintenance. Journal of Marketing Research, 45(6):633–644. find
  26. Mihalcea, Rada and Carlo Strapparava. 2009. The lie detector: Explorations in the automatic recognition of deceptive language. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 309–312, Suntec, Singapore.
  27. Mosteller, Frederick and David L. Wallace. 1963. Inference in an authorship problem. Journal of the American Statistical Association, 58(302):275–309. find
  28. Nakashole, Ndapandula and Tom M. Mitchell. 2014. Language-aware truth assessment of fact candidates. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1009–1019, Baltimore, MD. find
  29. Neal, Tempestt J., Kalaivani Sundararajan, Aneez Fatima, Yiming Yan, Yingfei Xiang, and Damon L. Woodard. 2017. Surveying stylometry techniques and applications. ACM Computing Surveys, 50:86:1–86:36. find
  30. Ott, Myle, Yejin Choi, Claire Cardie, and Jeffrey T. Hancock. 2011. Finding deceptive opinion spam by any stretch of the imagination. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 309–319, Portland, OR. Pérez-Rosas, Ve…
  31. Pisarevskaya, Dina. 2017. Deception detection in news reports in the Russian language: Lexics and discourse. In Proceedings of the 2017 EMNLP Workshop: Natural Language Processing meets Journalism, pages 74–79, Copenhagen. find
  32. Popat, Kashyap, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. 2018. DeClarE: Debunking fake news and false claims using evidence-aware deep learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 22–32, Brussels.
  33. Potthast, Martin, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. A stylometric inquiry into hyperpartisan and fake news. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 231–240, Melbourne.
  34. Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). https://openai.com/blog/ better-language-models/. ↗
  35. Rashkin, Hannah, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931–2937, Copenhagen.
  36. Rudanko, Juhani. 1982. Towards a description of negatively conditioned subject operator inversion in English. English Studies, 63(4):348–359. find
  37. Sari, Yunita, Andreas Vlachos, and Mark Stevenson. 2017. Continuous n-gram representations for authorship attribution. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 267–273, Valencia. find
  38. Schuster, Tal, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on…
  39. Shu, Kai, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social media: A data mining perspective. ACM SIGKDD Explorations Newsletter, 19(1):22–36. find
  40. Thompson, Andrew. 2017. All the news dataset. https://www.kaggle.com/ snapcrack/all-the-news. Kaggle. ↗
  41. Thorne, James, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Evaluating adversarial attacks against multiple fact verification systems. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu… find
  42. Thorne, James, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018. The fact extraction and verification (fever) shared task. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 1–9, Brussels.
  43. Trischler, Adam, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200, Vancouver.
  44. Tschiatschek, Sebastian, Adish Singla, Manuel Gomez Rodriguez, Arpit Merchant, and Andreas Krause. 2018. Fake news detection in social networks via crowd signals. In Companion Proceedings of the The Web Conference 2018, pages 517–524. find
  45. Tweedie, Fiona J., Shreyan Singh, and David I. Holmes. 1996. Neural network applications in stylometry: The Federalist Papers. Computers and the Humanities, 30:1–10. find
  46. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
  47. Volkova, Svitlana, Kyle Shaffer, Jin Yea Jang, and Nathan Hodas. 2017. Separating facts from fiction: Linguistic models to classify suspicious and trusted news posts on Twitter. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages… find
  48. Vosoughi, Soroush, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science, 359(6380):1146–1151. find
  49. Wang, William Yang. 2017. “Liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426, Vancouver.
  50. Wardle, Claire and Hossein Derakhshan. 2017. Information disorder: Toward an interdisciplinary framework for research and policy making. https://tinyurl.com/sy64l6s. ↗
  51. Wolf, Thomas, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
  52. Zellers, Rowan, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems 32, pages 9054–9065, Curran Associates, Inc.
  53. Zhao, Zhe, Paul Resnick, and Qiaozhu Mei. 2015. Enquiring minds: Early detection of rumors in social media from enquiry posts. In Proceedings of the 24th International Conference on World Wide Web, pages 1395–1405, Florence. find
  54. Zhou, Xinyi and Reza Zafarani. 2018. Fake news: A survey of research, detection methods, and opportunities. arXiv preprint arXiv:1812.00315. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schuster-2020-limitations-stylometry-detecting-machine-generated/