Elamur Research⌕
☰
home / papers / 2020 / TweepFake: about Detecting Deepfake Tweets

TweepFake: about Detecting Deepfake Tweets

Paper · 2020 · Tiziano Fagni, Fabrizio Falchi, Margherita Gambini et al.

Summary

Modern generators such as GPT-2 can produce convincing text, so social bots could flood platforms with machine-written posts. Earlier detection work focused mostly on long, purpose-generated texts like news articles, usually produced by one known generator. This left open whether detectors work on short messages actually published online by unknown models. Fagni et al. address this gap with TweepFake, which they describe as the first labeled dataset of real deepfake tweets. Every bot-written tweet in it was genuinely posted on Twitter. The bots use several generation approaches, and their tweets are balanced against tweets from the human accounts they imitate. The authors then benchmark 13 detectors of three kinds:

  • classic classifiers on text representations,
  • deep networks trained from scratch,
  • fine-tuned transformers.

Fine-tuned RoBERTa performs best overall, at about 90% accuracy. However, every method struggles with GPT-2 tweets, where a character-level GRU outperforms the transformers. The results show that transformer-generated short texts are hard to unmask. The dataset and code give future social-media detection research a public baseline.

Key points

  • TweepFake contents: 25,572 tweets split evenly between humans and bots, collected from 17 human accounts and the 23 bots that mimic them. These bots rely on Markov Chains, RNN, RNN+Markov, LSTM and GPT-2 for generation. The dataset is on Kaggle and the experiment code is on GitHub.
  • Accuracy by representation type:
    • BoW-based methods: about 0.80
    • BERT-based features: about 0.83
    • character encodings: up to 0.85
    • fine-tuned RoBERTa: about 0.90
  • Error patterns: All detectors had higher precision on human tweets and higher recall on bot tweets. Fine-tuned models (except XLNet) were the most balanced between the two.
  • Hardest category: GPT-2 tweets were the most difficult to detect, while RNN and "others" bots were caught easily. CHAR GRU beat the fine-tuned language models on GPT-2 tweets.
  • Practical note: Character-level deep models are simple and need no pretrained model. This makes them an option for languages lacking such resources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TrentoTrento · Italy87 works · 15,761 citations🎓University of PisaPisa · Italy26 works · 4,076 citations🔬Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo"Pisa · Italy5 works · 6,054 citations🔬Institute of Informatics and TelematicsPisa · Italy4 works · 485 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksInformation Systems · 1,730 worksSociology and Political Science · 626 works

Cite

Cite as Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella and Maurizio Tesconi. 2020. TweepFake: about Detecting Deepfake Tweets. arXiv:2008.00036.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1371/journal.pone.0251415 · arXiv 2008.00036

@misc{fagni2020tweepfake,
  title = {{TweepFake: about Detecting Deepfake Tweets}},
  author = {Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella and Maurizio Tesconi},
  year = {2020},
  doi = {10.1371/journal.pone.0251415},
  eprint = {2008.00036},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1371/journal.pone.0251415}
}

References

65 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vincent J. Why we need a better definition of ‘deepfake’. The Verge; 2018. Available from: https://www.theverge.com/2018/5/22/17380306/deepfake-definition-ai-manipulation-fake-news [cited 2021 March 09]. ↗
  2. Museum D. Behind the Scenes: Dali Lives.; 2019. In: Youtube [Video]. Available from: https://www.youtube.com/watch?v=BIDaxl4xqJ4&t=35s [cited 2021 March 09]. ↗
  3. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners; 2019. In: OpenAI Blog [Internet]. Available from: https://openai.com/blog/better-language-models/ [cited 2021 March 09]. ↗
  4. Gehrmann S, Strobelt H, Rush A. GLTR: Statistical Detection and Visualization of Generated Text. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics; 2019. p. 111–116.
  5. Adelani DI, Mai H, Fang F, Nguyen HH, Yamagishi J, Echizen I. Generating sentiment-preserving fake online reviews using neural language models and their human-and machine-based detection. In: International Conference on Advanced Information Networking and Applications. Springer; 2019. p. 1341–1354. find
  6. Tolosana R, Vera-Rodriguez R, Fierrez J, Morales A, Ortega-Garcia J. DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection. Information Fusion. 2020;64:131–148. find
  7. Lyu S. Deepfake Detection: Current Challenges and Next Steps. In: 2020 IEEE International Conference on Multimedia Expo Workshops (ICMEW); 2020. p. 1–6. find
  8. Nguyen TT, Nguyen CM, Nguyen DT, Nguyen DT, Nahavandi S. Deep Learning for Deepfakes Creation and Detection; 2019. arXiv:1909.11573 [Preprint]. Available from: https://arxiv.org/abs/1909.11573 [cited 2021 March 09]. — [arXiv:1909.11573](https://arxiv.org/abs/1909.11573) ↗
  9. Chen T, Kumar A, Nagarsheth P, Sivaraman G, Khoury E. Generalization of Audio Deepfake Detection. In: Proc. Odyssey 2020 The Speaker and Language Recognition Workshop; 2020. p. 132–137. find
  10. Wolff M, Wolff S. Attacking Neural Text Detectors; 2020. arXiv:2002.11768 [Preprint]. Available from: https://ui.adsabs.harvard.edu/abs/2020arXiv200211768W [cited 2021 March 09]. — [arXiv:2002.11768](https://arxiv.org/abs/2002.11768) ↗
  11. Bakhtin A, Gross S, Ott M, Deng Y, Ranzato M, Szlam A. Real or fake? learning to discriminate machine from human generated text; 2019. arXiv:1906.03351 [Preprint]. Available from: https://arxiv.org/abs/1906.03351 [cited 2021 March 09]. — [arXiv:1906.03351](https://arxiv.org/abs/1906.03351) ↗
  12. Ippolito D, Duckworth D, Callison-Burch C, Eck D. Automatic Detection of Generated Text is Easiest when Humans are Fooled. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics; 2020. p. 1808–1822.
  13. Garcia-Silva A, Berrio C, Gómez-Pérez JM. An Empirical Study on Pre-trained Embeddings and Language Models for Bot Detection. In: Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019). Association for Computational Linguistics; 2019. p. 148–155. find
  14. Lundberg J, Nordqvist J, Matosevic A. On-the-fly Detection of Autogenerated Tweets; 2018. arXiv:1802.01197 [Preprint]. Available from: https://arxiv.org/pdf/1802.01197.pdf [cited 2021 March 09]. — [arXiv:1802.01197](https://arxiv.org/abs/1802.01197) ↗
  15. Gorwa R, Guilbeault D. Unpacking the social media bot: A typology to guide research and policy. In: Special Issue:Cleaning Up Social Media. vol. 2. Policy & Internet; 2020. p. 225–248. find
  16. Cresci S, Di Pietro R, Petrocchi M, Spognardi A, Tesconi M. The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In: Proceedings of the 26th international conference on world wide web companion. International World Wide Web Conferences Steering Committee; 2017. p.… find
  17. Fagni T, Falchi F, Gambini M, Martella A, Tesconi M. TweepFake dataset; 2020. Repository: Kaggle [Internet]. Available from: https://www.kaggle.com/mtesconi/twitter-deep-fake-text [cited 2021 March 09]. ↗
  18. Fagni T. Scripts used in “TweepFake: about Detecting Deepfake Tweets”; 2021. Repository: GitHub [Internet]. Available from: https://github.com/tizfa/tweepfake_deepfake_text_detection [cited 2021 March 16]. ↗
  19. Bregler C, Covell M, Slaney M. Video rewrite: Driving visual speech with audio. In: Proceedings of the 24th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co.; 1997. p. 353–360. find
  20. Suwajanakorn S, Seitz SM, Kemelmacher-Shlizerman I. Synthesizing Obama: Learning Lip Sync from Audio. ACM Trans Graph. 2017;doi:10.1145/3072959.3073640. — [doi:10.1145/3072959.3073640](https://doi.org/10.1145/3072959.3073640) find
  21. Thies J, Zollhöfer M, Stamminger M, Theobalt C, Nießner M. Face2Face: Real-Time Face Capture and Reenactment of RGB Videos. Commun ACM. 2018;62(1):96–104. find
  22. Chan C, Ginosar S, Zhou T, Efros AA. Everybody dance now. In: Proceedings of the IEEE International Conference on Computer Vision. IEEE; 2019. p. 5933–5942. find
  23. Lyons K. FTC says the tech behind audio deepfakes is getting better. The Verge; 2020. Available from: https://www.theverge.com/2020/1/29/21080553/ftc-deepfakes-audio-cloning-joe-rogan-phone-scams [cited 2021 March 09]. ↗
  24. Jia Y, Zhang Y, Weiss R, Wang Q, Shen J, Ren F, et al. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. Curran Associates Inc.; 2018. p. 4480–4490. find
  25. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. Curran Associates Inc.; 2017. p. 6000–6010.
  26. Radford A, Narasimhan K, Salimans T, Sutskever I. Improving language understanding by generative pre-training; 2018. In: OpenAI Blog [Internet]. Available from: https://openai.com/blog/language-unsupervised/ [cited 2021 March 09]. ↗
  27. Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. vol. 1. Association for Comput…
  28. Zellers R, Holtzman A, Rashkin H, Bisk Y, Farhadi A, Roesner F, et al. Defending Against Neural Fake News. In: Wallach H, Larochelle H, Beygelzimer A, d'Alché-Buc F, Fox E, Garnett R, editors. Advances in Neural Information Processing Systems 32. Curran Associates, Inc.; 2019. p. 9054–9065.
  29. Keskar NS, McCann B, Varshney LR, Xiong C, Socher R. Ctrl: A conditional transformer language model for controllable generation; 2019. arXiv:1909.05858 [Preprint]. Available from: https://arxiv.org/abs/1909.05858 [cited 2021 March 09]. ↗
  30. Li C, Gao X, Li Y, Li X, Peng B, Zhang Y, et al. Optimus: Organizing sentences via pre-trained modeling of a latent space. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics; 2020. p. 4678–4699.
  31. Solaiman I, Brundage M, Clark J, Askell A, Herbert-Voss A, Wu J, et al.. Release strategies and the social impacts of language models; 2019. arXiv:1908.09203 [Preprint]. Available from: https://arxiv.org/abs/1908.09203 [cited 2021 March 09]. ↗
  32. Platen PV. How to generate text: using different decoding methods for language generation with Transformers.; 2020. In: Huggingface Blog [Internet]. Available from: https://huggingface.co/blog/how-to-generate [cited 2021 March 09]. ↗
  33. Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov RR, Le QV. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In: Wallach H, Larochelle H, Beygelzimer A, d'Alché-Buc F, Fox E, Garnett R, editors. Advances in Neural Information Processing Systems. vol. 32. Curran Associates,…
  34. Mann B. How to sample from language models; 2019. In: Towards Data Science [Internet]. Available from: https://towardsdatascience.com/how-to-sample-from-language-models-682bceb97277 [cited 2021 March 09]. ↗
  35. Macbeth S. Whalefakes code; 2019. Repository: Github [Internet]. Available from: https://github.com/sorenmacbeth/gpt-2-whalefakes [cited 2021 March 09]. ↗
  36. Woolf M. How To Make Custom AI-Generated Text With GPT-2; 2019. In: Max Woolf’s blog [Internet]. Available from: https://minimaxir.com/2019/09/howto-gpt2/ [cited 2021 March 09]. ↗
  37. Woolf M. Train a GPT-2 Text-Generating Model w/ GPU; 2020. In: Google Colab Notebook [Internet]. Available from: https://colab.research.google.com/drive/1VLG8e7YSEwypxU-noRNhsv5dW4NfTGce [cited 2021 March 09]. ↗
  38. Cohn B. drilbot: GPT-2 finetuned on dril tweets; 2019. Repository: Github [Internet]. Available from: https://github.com/o1brad/drilbot [cited 2021 March 09]. ↗
  39. Woolf M. twitter-cloud-run; 2020. Repository: Github [Internet]. Available from: https://github.com/minimaxir/twitter-cloud-run [cited 2021 March 09]. ↗
  40. Woolf M. botustrump; 2020. Repository: Github [Internet]. Available from: https://github.com/osirisguitar/botus-twitter [cited 2021 March 09]. ↗
  41. Hooke K. Generating Tweets Using a Recurrent Neural Net (torch-rnn); 2018. In: DZone [Internet]. Available from: https://dzone.com/articles/generating-tweets-using-a-recurrent-neural-net-tor [cited 2021 March 09]. ↗
  42. Aggarwal R. Develop and publish Text Generating Twitter bot; 2019. In: Medium [Internet]. Available from: https://medium.com/@aggarwal.rohan.me/develop-and-publish-text-generating-twitter-bot-9da60cfd20b8 [cited 2021 March 09]. ↗
  43. Aggarwal R. AI-NarendraModi-twitter-Bot; 2019. Repository: Github [Internet]. Available from: https://github.com/aggrowal/AI-NarendraModi-twitter-bot [cited 2021 March 09]. ↗
  44. Hayes B. Postdoc develops Twitterbot that uses AI to sound like Donald Trump; 2016. In: MIT’s blog [Internet]. Available from: https://www.csail.mit.edu/news/postdoc-develops-twitterbot-uses-ai-sound-donald-trump [cited 2021 March 09]. ↗
  45. Hayes B. DeepDrumpf; 2016. Repository: Github [Internet]. Available from: https://github.com/deepdrumpf/deepdrumpf.github.io [cited 2021 March 09]. ↗
  46. Deep Elon: AI Tweets from Mars; 2018. In: Twitter [Internet]. Available from: https://twitter.com/musk_from_mars [cited 2021 March 09]. ↗
  47. Smith D. deep-elon-tweet-generator; 2018. Repository: Github [Internet]. Available from: https://github.com/DaveSmith227/deep-elon-tweet-generator [cited 2021 March 09]. ↗
  48. Meyer S. Create a Twitter Politician Bot with Markov Chains, Node.js and StdLib; 2017. In: Hackernoon [Internet]. Available from: https://hackernoon.com/create-a-twitter-politician-bot-with-markov-chains-node-js-and-stdlib-14df8cc1c68a [cited 2021 March 09]. ↗
  49. Meyer S. The Jaden Trudeau Code: twitter-markov-chain;. In: Autocode [Internet]. Available from: https://autocode.com/lib/steve/twitter-markov-chain/ [cited 2021 March 09]. ↗
  50. Deep Elon Musk: IA entrainée pour tweeter comme Elon Musk; 2017. In: OpenClassrooms [Internet]. Available from: https://openclassrooms.com/forum/sujet/ia-deep-elon-musk [cited 2021 March 09]. ↗
  51. Elon Musk Transcripts Dataset; 2017. Repository: Github [Internet]. Available from: https://github.com/berpj/elon-musk-dataset [cited 2021 March 09]. ↗
  52. Mete J. deepThorin; 2017. Repository: Github [Internet]. Available from: https://github.com/Jmete/deepThorin [cited 2021 March 09]. ↗
  53. Sebastiani F. Machine Learning in Automated Text Categorization. ACM Computing Surveys. 2002;34(1):1–47.
  54. Nigam K, Lafferty J, McCallum A. Using maximum entropy for text classification. In: IJCAI-99 workshop on machine learning for information filtering. vol. 1. Stockholom, Sweden; 1999. p. 61–67. find
  55. Breiman L. Random Forests. Mach Learn. 2001;doi:10.1023/A:1010933404324. find
  56. Joachims T. Text categorization with support vector machines: Learning with many relevant features. In: Nédellec C, Rouveirol C, editors. Machine Learning: ECML-98. Springer; 1998. p. 137–142.
  57. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE. 1998;86(11):2278–2324. find
  58. Cho K, van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, et al. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computat…
  59. Verleysen M, François D. The curse of dimensionality in data mining and time series prediction. In: International Work-Conference on Artificial Neural Networks. Springer; 2005. p. 758–770. find
  60. Zhang X, Zhao J, LeCun Y. Character-level Convolutional Networks for Text Classification. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R, editors. Advances in Neural Information Processing Systems 28. Curran Associates, Inc.; 2015. p. 649–657.
  61. Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, et al.. Roberta: A robustly optimized bert pretraining approach; 2019. arXiv:1907.11692 [Preprint]. Available from: https://arxiv.org/abs/1907.11692 [cited 2021 March 09]. ↗
  62. Buitinck L, Louppe G, Blondel M, Pedregosa F, Mueller A, Grisel O, et al. API design for machine learning software: experiences from the scikit-learn project. In: ECML PKDD Workshop: Languages for Data Mining and Machine Learning; 2013. p. 108–122. find
  63. Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et al. Transformers: State-of-the-Art Natural Language Processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics; 2020. p. 38–45.
  64. Chollet F. Keras; 2015. Repository: Github [Internet]. Available from: https://github.com/keras-team/keras [cited 2021 March 09]. ↗
  65. Rajapakse TC. Simple Transformers; 2019. Repository: Github [Internet]. Available from: https://github.com/ThilinaRajapakse/simpletransformers [cited 2021 March 09]. Fig 3. Supp. info 1: Parameter values. Parameter values used in the final experimentation on the test set. Fig 4. Supp. info 2: Acc. o… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fagni-2020-tweepfake-about-detecting-deepfake-tweets/