Elamur Research⌕
☰
home / papers / 2021 / Arabic Offensive Language on Twitter: Analysis and Experiments

Arabic Offensive Language on Twitter: Analysis and Experiments

Summary

Only about 1–2% of Arabic tweets are offensive, so annotating random samples to build training data is very inefficient. Filtering with a seed list of offensive words would raise the yield but skew the data toward particular dialects, topics or targets. The authors avoid this by selecting tweets that contain the Arabic vocative particle "yA" ("O") at least twice. People use this construction to address someone directly, and it appears in every dialect and on any subject. From 660k matching tweets, they kept one random tweet per frequent unique word sequence between the particles and annotated 10,000 tweets. Labels follow guidelines compatible with OffensEval2019: offensive, with optional vulgar and/or hate speech sub-tags, or clean. About 19% of the resulting set is offensive. The paper describes this as the largest Arabic offensive-language dataset to date, and the authors also analyse its topics, dialects and author gender. In classification experiments, fine-tuned AraBERT performed best (F1 = 83.2), with Mazajak embeddings plus an SVM close behind. The resource gives Arabic offensive-language detection a sizeable benchmark that is less biased by dialect or topic.

Key points

  • Data collection

    • 660k Arabic tweets with the "yA … yA" pattern were gathered via the Twitter API between April 15 and May 6, 2019.
    • The 10,000 most frequent unique in-between word sequences were kept, with one random tweet per sequence.
    • 1,915 of these tweets (~19%) were labeled offensive.

  • Yield of the vocative filter: the offensive share rises from 1–2% in random tweets to about 5% with one vocative particle and about 19% with two or more.

  • Label scheme:

    • The labels are OFF, VLG, HS and CLN, and vulgarity and hate speech can overlap.
    • Insults aimed at sports-club fans count as hate speech, alongside insults based on race, ethnicity, political group and religion.

  • Classification results (5-fold cross-validation, identical folds for all runs):

    ModelF1
    AraBERT83.2
    Mazajak/SVM79.7
    AraVec/SVM76.4
    BERT base-multilingual76.0
    fastText trained on the data74.4
    Lexical-feature SVM46.6

    An RBF-kernel SVM beat seven other classifiers when using Mazajak embeddings.

  • Error analysis (one fold, Mazajak/SVM: 36 false positives, 121 false negatives):

    • False positives came from gloating, quoting someone else's insult, idioms and sarcasm.
    • False negatives came from insults mixed with admiration and from implicit offensiveness.

Interesting ideas

  • I1. Use a neutral structural cue to oversample a rare class. Instead of a lexicon of bad words, filter on a grammatical marker associated with addressing someone, which is independent of topic and dialect. Any rare-phenomenon annotation project could look for similar markers to raise yield without baking a keyword list's biases into the data.
  • I2. Deduplicate by pattern content to increase diversity. Keeping one tweet per unique word sequence between the particles stops a few popular phrases from dominating the sample. This is a cheap diversity trick for any pattern-mined corpus.
  • I3. Use hierarchical, overlapping labels. Treating vulgarity and hate speech as non-exclusive sub-tags of "offensive" keeps one simple binary task while still allowing finer analyses. This suits taxonomies where categories genuinely co-occur.
  • I4. Adapt guideline categories to the culture. Counting attacks on sports-club fans as hate speech reflects how these identities work in the target culture. It is a reminder to revise imported annotation guidelines to fit local social realities.
  • I5. Weigh domain match against model size, and run a typed error analysis. In-domain static embeddings with an SVM outperformed multilingual BERT, which suggests the data domain can matter more than model complexity. Grouping errors into types such as quoting, idioms and sarcasm also points to where context-aware modeling is needed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

82 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 72 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mubarak2021arabic,
  title = {{Arabic Offensive Language on Twitter: Analysis and Experiments}},
  author = {Mubarak, Hamdy and Rashed, Ammar and Darwish, Kareem and Samih, Younes and Abdelali, Ahmed},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.13/}
}

References

38 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16. find our card
  2. Ehab Abozinadah. 2017. Detecting Abusive Arabic Language Twitter Accounts Using a Multidimensional Analysis Model. Ph.D. thesis, George Mason University. find
  3. Ibrahim Abu Farha and Walid Magdy. 2019. Mazajak: An online Arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy. Association for Computational Linguistics. find
  4. Sweta Agrawal and Amit Awekar. 2018. Deep learning for detecting cyberbullying across multiple social media platforms. In European Conference on Information Retrieval, pages 141–153. Springer. find
  5. Azalden Alakrot, Liam Murray, and Nikola S Nikolov. 2018. Towards accurate detection of offensive language in online communication in arabic. Procedia computer science, 142:315–320. find
  6. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2018. Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. IEEE. find
  7. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. find our card
  8. Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. Deep learning for hate speech detection in tweets. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 759–760. International World Wide Web Conferences Steering Committee. find
  9. Pablo Barberá and Gaurav Sood. 2015. Follow your ideology: Measuring media ideology on social networks. In Annual Meeting of the European Political Science Association, Vienna, Austria. Retrieved from http://www.gsood. com/research/papers/mediabias. pdf. ↗
  10. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗
  11. Kareem Darwish, Dimitar Alexandrov, Preslav Nakov, and Yelena Mejova. 2017. Seminar users in the arabic twitter sphere. In International Conference on Social Informatics, pages 91–108. Springer. find
  12. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Eleventh International Conference on Web and Social Media (ICWSM), pages 512–515. find our card
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  14. Nemanja Djuric, Jing Zhou, Robin Morris, Mihajlo Grbovic, Vladan Radosavljevic, and Narayan Bhamidipati. 2015. Hate speech detection with comment embeddings. In Proceedings of the 24th international conference on world wide web, pages 29–30. ACM. find
  15. Samhaa R. El-Beltagy. 2016. NileULex: A phrase and word level sentiment lexicon for Egyptian and modern standard Arabic. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2900–2905, Portorož, Slovenia. European Language Resources Association (ELRA). find
  16. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. ↗
  17. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia. Association for Computational Linguistics. find
  18. Timothy Jay and Kristin Janschewitz. 2008. The pragmatics of swearing. Journal of Politeness Research. Language, Behaviour, Culture, 4(2):267–288. find
  19. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431. Association for Computational Linguistics. find
  20. Irene Kwok and Yuzhou Wang. 2013. Locate the hate: Detecting tweets against blacks. In Twenty-seventh AAAI conference on artificial intelligence. find
  21. Thomas Chadefaux. 2014. Early warning signals for war in the news. Journal of Peace Research, 51(1):5–18. find
  22. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. find our card
  23. Michael Conover, Jacob Ratkiewicz, Matthew R Francisco, Bruno Gonçalves, Filippo Menczer, and Alessandro Flammini. 2011. Political polarization on twitter. ICWSM, 133:89–96. find
  24. Alessandro Maisto, Serena Pelosi, Simonetta Vietri, Pierluigi Vitale, and Via Giovanni Paolo II. 2017. Mining offensive language on social media. CLiC-it 2017 11-12 December 2017, Rome, page 252. find
  25. Shervin Malmasi and Marcos Zampieri. 2017. Detecting hate speech in social media. arXiv preprint arXiv:1712.06427. ↗
  26. Elisa Mattiello. 2005. The pervasiveness of slang in standard and non-standard english. Mots Palabras Words, 5:7–41. find
  27. Abu Bakr Mohammad, Kareem Eissa, and Samhaa El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  28. Hamdy Mubarak and Kareem Darwish. 2019. Arabic offensive language classification on twitter. In International Conference on Social Informatics, pages 269–276. Springer. find
  29. Hamdy Mubarak, Kareem Darwish, and Walid Magdy. 2017. Abusive language detection on arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56. find our card
  30. Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive language detection in online user content. In Proceedings of the 25th international conference on world wide web, pages 145–153. International World Wide Web Conferences Steering Committee. find
  31. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf. ↗
  32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. find our card
  33. Younes Samih, Mohamed Eldesouki, Mohammed Attia, Kareem Darwish, Ahmed Abdelali, Hamdy Mubarak, and Laura Kallmeyer. 2017. Learning from relatives: unified dialectal arabic segmentation. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 432–441. find
  34. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. find our card
  35. Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88–93. find our card
  36. Dawei Yin, Zhenzhen Xue, Liangjie Hong, Brian D Davison, April Kontostathis, and Lynne Edwards. 2009. Detection of harassment on web 2.0. Proceedings of the Content Analysis in the WEB, 2:1–7. find
  37. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). arXiv preprint arXiv:1903.08983. ↗ our card
  38. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020). arXiv preprint arXiv:2006.07235. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mubarak-2021-arabic-offensive-language-twitter-analysis/