Elamur Research⌕
☰
home / papers / 2020 / Bert Transformer model for Detecting Arabic GPT2 Auto-Generated Tweets

Bert Transformer model for Detecting Arabic GPT2 Auto-Generated Tweets

Paper · 2020 · Fouzi Harrag, Maria Dabbah, Kareem Darwish et al.

Summary

The paper tackles a classification problem for Arabic social media: telling apart tweets written by people from "deepfake" text produced by a neural language model. No suitable resource existed, so the authors built one. They took the human-labelled tweets from Almerekhi and Elsayed (2015) and enlarged that pool by crawling the timelines of the same accounts. They then used GPT2-Small-Arabic, prompted with those human tweets, to produce a matching machine-written set. The previous bot dataset was set aside as a negative class because much of its bot content consisted of Quran verses, proverbs and sexual spam, which does not fit the idea of human-like generated text. For detection, the authors fine-tuned AraBERT (v0.1) and compared it with four recurrent baselines built on randomly initialised word embeddings. The fine-tuned transformer clearly beat all RNNs, reaching about 98.7% accuracy. The work matters because it is presented as the first pairing of GPT2-style generation with AraBERT detection for Arabic. It also supplies a template for building generated-vs-human corpora in lower-resource languages.

Key points

  • Human side: 1,559 "human content" tweets from Almerekhi and Elsayed (2015), expanded through timeline crawling to 4,196 tweets. Deleted tweets and suspended accounts were dropped.
  • Machine side: 3,512 texts of 15–35 words, generated by GPT2-Small-Arabic seeded with the human tweets. That model was trained on roughly 900MB of Arabic Wikipedia with Fastai2.
  • Preprocessing steps:
    • stripped URLs, non-Arabic characters, punctuation and diacritics;
    • split hashtags;
    • replaced user mentions with a USER token.
  • Experimental setup and baselines:
    • the data was split randomly into 80% train and 20% test;
    • the baselines were LSTM, GRU, biLSTM and biGRU, each with 50-dimensional embeddings, an RNN layer of 128 units (256 when bidirectional) and a 400-unit dense layer;
    • biLSTM was the strongest baseline, with 96.3 accuracy and 96.0 F1.
  • AraBERT was fine-tuned for 4 epochs with HuggingFace PyTorch. It scored 98.7 accuracy, 98.9 precision, 98.5 recall and 98.7 F1.

Interesting ideas

  • I1. Seed the generator with each real human text so the machine-written class matches the human class in topic and register, giving a paired contrast instead of an unrelated negative set.
  • I2. Before reusing an existing labelled dataset, the authors checked whether its "positive" examples really fit the target definition, and dropped the bot class when they found mostly scripture, proverbs and spam.
  • I3. Labels were extended cheaply by inheriting them from the source: tweets from accounts already known to be human were assumed human, and this simplifying assumption was stated openly.
  • I4. The authors suspect their near-perfect detector may simply be exploiting repetition and the stylistic habits of generated text, and they plan explainability tools to check this.
  • I5. Normalising social text by swapping mentions for a USER placeholder and splitting hashtags keeps the vocabulary small without losing sentence structure.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Mova's text → graph → text compressor output, paired with its source sentences, could feed a simple discriminator, and how easily that discriminator separates them would score how human-like the regenerated text is.
  • I2 (learn): silver data borrowed for a task such as irony or speech acts should first be spot-checked against Mova's own definition of the label, so that look-alike categories do not leak into training.
  • I3 (learn): silver labels could be propagated at document or source level, with the assumption recorded per example so that later repair rules can find and revisit those labels.
  • I4 (learn): any high score should be compared against a trivial surface-cue control, such as a repetition-only feature, to confirm that a component is actually using structure.
  • I5 (tokens): the en tokenizer could map mentions and URLs to typed placeholder tokens and split hashtags into words, so the tagger and parser see clean, parseable input from social-media sources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Fouzi Harrag, Maria Dabbah, Kareem Darwish and Ahmed Abdelali. 2020. Bert Transformer model for Detecting Arabic GPT2 Auto-Generated Tweets. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{harrag2020bert,
  title = {{Bert Transformer model for Detecting Arabic GPT2 Auto-Generated Tweets}},
  author = {Fouzi Harrag, Maria Dabbah, Kareem Darwish and Ahmed Abdelali},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.19/}
}

References

19 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Khooli Abed. 2020. huggingface , gpt2-small-arabic. https://huggingface.co/akhooli/ gpt2-small-arabic. Available Online: 2019-01-15, Accessed: 2020-09-30. ↗
  2. Fatih Cagatay Akyon and M Esat Kalfaoglu. 2019. Instagram fake and automated account detection. In 2019 Innovations in Intelligent Systems and Applications Conference (ASYU), pages 1–7. IEEE. find
  3. Hind Almerekhi and Tamer Elsayed. 2015. Detecting automatically-generated arabic tweets. In AIRS, pages 123–134. Springer. find
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  5. Chiyu Cai, Linjing Li, and Daniel Zengi. 2017. Behavior enhanced deep bot detection in social media. In 2017 IEEE International Conference on Intelligence and Security Informatics (ISI), pages 128–130. IEEE. find
  6. Kareem Darwish, Dimitar Alexandrov, Preslav Nakov, and Yelena Mejova. 2017. Seminar users in the arabic twitter sphere. In International Conference on Social Informatics, pages 91–108. Springer. ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a. Bert: Pre-training of deep bidirectional transformers for language understanding.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  9. Numa Dhamani, Paul Azunre, Jeffrey L. Gleason, Craig Corcoran, Garrett Honke, Steve Kramer, and Jonathon Morgan. 2019. Using deep networks and transfer learning to address disinformation. ↗
  10. Touseef Iqbal and Shaima Qureshi. 2020. The survey: Text generation models in deep learning. Journal of King Saud University - Computer and Information Sciences. ↗
  11. Sneha Kudugunta and Emilio Ferrara. 2018. Deep neural networks for bot detection. Information Sciences, 467:312–322. ↗
  12. Edward Ma. 2019. towards data science, too powerful nlp model (gpt-2), what is generative pre-training? https://towardsdatascience.com/ too-powerful-nlp-model-generative-pre-training-2-4cc6afb6655. Available Online: 2019-02-21, Accessed: 2020-09-30. ↗
  13. Damian Radcliffe and Payton Bruni. 2019. State of Social Media Middle East: 2018. University of Oregon Libraries. ↗
  14. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  15. Giovanni C Santia, Munif Ishad Mujib, and Jake Ryland Williams. 2019. Detecting social bots on facebook in an information veracity context. In Proceedings of the International AAAI Conference on Web and Social Media, volume 13, pages 463–472. find
  16. Sima Siami-Namini, Neda Tavakoli, and Akbar Siami Namin. 2019. The performance of lstm and bilstm in forecasting time series. In 2019 IEEE International Conference on Big Data (Big Data), pages 3285–3292. IEEE. find
  17. Amirsina Torfi, Rouzbeh A. Shirvani, Yaser Keneshloo, Nader Tavaf, and Edward A. Fox. 2020. Natural language processing advancements by deep learning: A survey. ↗
  18. Onur Varol, Emilio Ferrara, Clayton A Davis, Filippo Menczer, and Alessandro Flammini. 2017. Online human-bot interactions: Detection, estimation, and characterization. arXiv preprint arXiv:1703.03107. ↗
  19. Feng Wei and Uyen Trang Nguyen. 2019. Twitter bot detection using bidirectional long short-term memory neural networks and word embeddings. In 2019 First IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA), pages 101–109. IEEE. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/harrag-2020-bert-transformer-model-detecting-arabic/