Elamur Research⌕
☰
home / papers / 2020 / Hate Speech Detection in Saudi Twittersphere: A Deep Learning Approach

Hate Speech Detection in Saudi Twittersphere: A Deep Learning Approach

Paper · 2020 · Raghad Alshaalan and Hend Al-Khalifa

Summary

The paper tackles automatic hate speech detection in Arabic tweets, with a focus on Saudi users. Little prior Arabic work exists, and the language is hard for this task because of rich morphology, varied orthography and many dialects. The authors built a new public corpus of 9,316 tweets, each labeled hateful, abusive or normal. To collect it, they searched with neutral terms naming groups that are often targeted, such as tribes, regions, religious sects and ideologies, and with hashtags on contentious topics. They then trained and compared four classifiers: CNN, GRU, a CNN+GRU hybrid and a fine-tuned BERT. These were tested on their own data and on a second, out-of-domain dataset. The plain CNN performed best (F1 0.79, AUROC 0.89). BERT did not beat the baselines, which the authors blame on its Wikipedia pretraining being far from tweet language. An error analysis shows that most missed hate contains no offensive words and depends on context. This makes the corpus and the failure study useful beyond Arabic.

Key points

  • Collection ran from March to August 2018 through the Twitter API (tweepy). It used 164 neutral group-referring terms, 10 trending controversial hashtags, and top terms from the religious hate lexicon of Albadi et al. (2018). This yielded about 54 million tweets, of which 10,000 were sampled for annotation on Figure Eight.
  • The final released corpus has 9,316 tweets labeled hateful, abusive or normal. It covers racist (regional, tribal), inter-religious and ideological hate.
  • Models compared were CNN, GRU, CNN+GRU and BERT (pretrained model plus a small task layer, fully fine-tuned). The CNN was best, with F1 0.79 and AUROC 0.89, and BERT did not improve over the baselines.
  • Error analysis covered hateful tweets missed by every model: 151 in GHSD and 51 in RHSD. Roughly 90% of these contained no abusive terms.
  • On the other side, 33 (GHSD) and 20 (RHSD) normal tweets were wrongly flagged. Nearly all of them mentioned religious, national or ideological affiliations that the authors' lexicons tie strongly to the hate class.

Interesting ideas

  • I1. Gather data with neutral words that name likely targets, not with slurs, so the resulting sample is not limited to openly offensive language; this reduces the bias of finding only what the query already encodes.
  • I2. Study only the items that every model gets wrong, since these shared failures expose the limits of the task rather than the weaknesses of one architecture.
  • I3. Normal texts that mention identity or affiliation terms were flagged as hateful, showing that the classifiers learned a lexical shortcut rather than meaning.
  • I4. Much missed hate was implicit, sarcastic, or meaningful only in light of a reply, a quoted tweet, or a vague "those" pointing to a group.
  • I5. A large pretrained model underperformed simple CNNs because its pretraining genre (Wikipedia) differed from noisy social text, which warns against assuming bigger is better under domain shift.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver-data and test-set sampling could seed on neutral topic or entity terms rather than on the constructions we expect, so evaluation sets include cases our rules were never tuned to.
  • I2 (learn): When several witness parsers or readers all fail on the same sentence, that intersection should become a dedicated bucket for repair-rule induction and LLM-teacher review.
  • I3 (learn): Build negative controls where trigger words appear in benign frames, to check that an answer depends on structure rather than on a word's presence.
  • I4 (frames): Irony and implicature detection should use the discourse antecedent (the replied-to or quoted text), and coreference should resolve group-denoting pronouns before stance is judged.
  • I5 (tree): When evaluating gparse, report results per genre against the perceptron parser, and add in-genre silver data where the transformer's pretraining domain diverges.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksBias, fairness & safety · 517 works

Cite

Cite as Raghad Alshaalan and Hend Al-Khalifa. 2020. Hate Speech Detection in Saudi Twittersphere: A Deep Learning Approach. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alshaalan2020hate,
  title = {{Hate Speech Detection in Saudi Twittersphere: A Deep Learning Approach}},
  author = {Raghad Alshaalan and Hend Al-Khalifa},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.2/}
}

References

38 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelali, A., Darwish, K., Durrani, N., & Mubarak, H. (2016). Farasa: A Fast and Furious Segmenter for Arabic. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, 11–16.
  2. Alakrot, A., Murray, L., & Nikolov, N. S. (2018). Towards Accurate Detection of Offensive Language in Online Communication in Arabic. Procedia Computer Science, 142, 315–320. https://doi.org/10.1016/j.procs.2018.10.491 ↗
  3. Albadi, N., Kurdi, M., & Mishra, S. (2018). Are they Our Brothers? Analysis and Detection of Religious Hate Speech in the Arabic Twittersphere. 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), 69–76. https://doi.org/10.1109/ASONAM.2018.8508247 ↗
  4. Al-Hassan, A., & Al-Dossari, H. (2019). DETECTION OF HATE SPEECH IN SOCIAL NETWORKS: A SURVEY ON MULTILINGUAL CORPUS. Computer Science & Information Technology(CS & IT), 83–100. https://doi.org/10.5121/csit.2019.90208 ↗
  5. Al-Humoud, S., Al-Twairesh, N., Altuwaijri, M., & Almoammar, A. (2016). Arabic Spam Detection in Twitter. find
  6. Al-Twairesh, N., & Al-Negheimish, H. (2019). Surface and Deep Features Ensemble for Sentiment Analysis of Arabic Tweets. IEEE Access, 7, 84122–84131. https://doi.org/10.1109/ACCESS.2019.2924314 ↗
  7. Badjatiya, P., Gupta, S., Gupta, M., & Varma, V. (2017). Deep Learning for Hate Speech Detection in Tweets. Proceedings of the 26th International Conference on World Wide Web Companion, 759–760. https://doi.org/10.1145/3041021.3054223 ↗
  8. Cer, D., Yang, Y., Kong, S., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., Sung, Y.-H., Strope, B., & Kurzweil, R. (2018). Universal Sentence Encoder. ArXiv:1803.11175 [Cs]. http://arxiv.org/abs/1803.11175 ↗
  9. Chen, H., McKeever, S., & Delany, S. J. (2018). A Comparison of Classical Versus Deep Learning Techniques for Abusive Content Detection on Social Media Sites. In S. Staab, O. Koltsova, & D. I. Ignatov (Eds.), Social Informatics (pp. 117–133). Springer International Publishing. find
  10. Çöltekin, Ç. (2020). A Corpus of Turkish Offensive Language on Social Media. Proceedings of The 12th Language Resources and Evaluation Conference. 2020., 6174–6184. https://www.aclweb.org/anthology/2020.lrec-1.758 ↗
  11. Dai, W., Yu, T., Liu, Z., & Fung, P. (2020). Kungfupanda at SemEval-2020 Task 12: BERT-Based Multi-Task Learning for Offensive Language Detection. ArXiv:2004.13432 [Cs]. http://arxiv.org/abs/2004.13432 ↗
  12. Darwish, K., Magdy, W., & Mourad, A. (2012). Language processing for arabic microblog retrieval. Proceedings of the 21st ACM International Conference on Information and Knowledge Management, 2427–2430. find
  13. Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017). Automated Hate Speech Detection and the Problem of Offensive Language. ArXiv:1703.04009 [Cs]. http://arxiv.org/abs/1703.04009 ↗
  14. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv:1810.04805 [Cs]. http://arxiv.org/abs/1810.04805 ↗
  15. Djuric, N., Zhou, J., Morris, R., Grbovic, M., Radosavljevic, V., & Bhamidipati, N. (2015). Hate Speech Detection with Comment Embeddings. Proceedings of the 24th International Conference on World Wide Web - WWW ’15 Companion, 29–30. https://doi.org/10.1145/2740908.2742760 ↗
  16. El Mahdaouy, A., El Alaoui, S. O., & Gaussier, E. (2018). Word-embedding-based pseudo-relevance feedback for Arabic information retrieval. Journal of Information Science, 016555151879221. https://doi.org/10.1177/0165551518792210 ↗
  17. Fortuna, P., & Nunes, S. (2018). A Survey on Automatic Detection of Hate Speech in Text. ACM Comput. Surv., 51(4), 85:1–85:30. https://doi.org/10.1145/3232676 ↗
  18. Gambäck, B., & Sikdar, U. K. (2017). Using Convolutional Neural Networks to Classify Hate-Speech. Proceedings of the First Workshop on Abusive Language Online, 85–90. https://doi.org/10.18653/v1/W17-3013 ↗
  19. Jaki, S., & De Smedt, T. (2018). Right-wing German hate speech on Twitter: Analysis and automatic detection. Manuscript Submitted. ↗
  20. Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2016). Bag of Tricks for Efficient Text Classification. ArXiv:1607.01759 [Cs]. http://arxiv.org/abs/1607.01759 ↗
  21. Kaji, N., & Kitsuregawa, M. (2007). Building Lexicon for Sentiment Analysis from Massive Collection of HTML Documents. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL. find
  22. Kim, Y. (2014). Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1746–1751. http://www.aclweb.org/anthology/D14-1181 ↗
  23. Kwok, I., & Wang, Y. (2013). Locate the hate: Detecting tweets against blacks. Twenty-Seventh AAAI Conference on Artificial Intelligence. find
  24. Magdy, W., Darwish, K., & Weber, I. (2015). #FailedRevolutions: Using Twitter to Study the Antecedents of ISIS Support. ArXiv:1503.02401 [Physics]. http://arxiv.org/abs/1503.02401 ↗
  25. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. ArXiv:1301.3781 [Cs]. http://arxiv.org/abs/1301.3781 ↗
  26. Mozafari, M., Farahbakhsh, R., & Crespi, N. (2020). A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media. In H. Cherifi, S. Gaito, J. F. Mendes, E. Moro, & L. M. Rocha (Eds.), Complex Networks and Their Applications VIII (pp. 928–940). Springer International Publishing. https://doi.org/10.1007/978-3-030-36687-2_77 ↗
  27. Mubarak, H., Darwish, K., & Magdy, W. (2017). Abusive Language Detection on Arabic Social Media. Proceedings of the First Workshop on Abusive Language Online, 52–56. http://www.aclweb.org/anthology/W17-3008 ↗
  28. Mubarak, H., Rashed, A., Darwish, K., Samih, Y., & Abdelali, A. (2020). Arabic Offensive Language on Twitter: Analysis and Experiments. ArXiv:2004.02192 [Cs]. http://arxiv.org/abs/2004.02192 ↗
  29. Mulki, H., Haddad, H., Bechikh Ali, C., & Alshabani, H. (2019). L-HSAB: A Levantine Twitter Dataset for Hate Speech and Abusive Language. Proceedings of the Third Workshop on Abusive Language Online, 111– 118. https://doi.org/10.18653/v1/W19-3512 ↗
  30. Nobata, C., Tetreault, J., Thomas, A., Mehdad, Y., & Chang, Y. (2016). Abusive Language Detection in Online User Content. Proceedings of the 25th International Conference on World Wide Web - WWW ’16, 145– 153. https://doi.org/10.1145/2872427.2883062 ↗
  31. Park, J. H., & Fung, P. (2017). One-step and Two-step Classification for Abusive Language Detection on Twitter. Proceedings of the First Workshop on Abusive Language Online, 41–45. https://doi.org/10.18653/v1/W17-3006 ↗
  32. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. ArXiv Preprint ArXiv:1802.05365. ↗
  33. Pitsilis, G. K., Ramampiaro, H., & Langseth, H. (2018). Effective hate-speech detection in Twitter data using recurrent neural networks. Applied Intelligence, 48(12), 4730–4742. https://doi.org/10.1007/s10489-018-1242-y ↗
  34. Schmidt, A., & Wiegand, M. (2017). A Survey on Hate Speech Detection using Natural Language Processing. Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, 1– 10. http://www.aclweb.org/anthology/W17-1101 ↗
  35. Sigurbergsson, G. I., & Derczynski, L. (n.d.). Offensive Language and Hate Speech Detection for Danish. Proceedings of The 12th Language Resources and Evaluation Conference, 3498–3508. Singh, A., Blanco, E., & Jin, W. (2019). Incorporating Emoji Descriptions Improves Tweet Classification. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2096– 2101. https://doi.org/10.18653/v1/N19-1214 ↗
  36. Warner, W., & Hirschberg, J. (2012). Detecting Hate Speech on the World Wide Web. Proceedings of the Second Workshop on Language in Social Media, 19–26. http://dl.acm.org/citation.cfm?id=2390374.2390377 ↗
  37. Waseem, Z., & Hovy, D. (2016). Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. Proceedings of the NAACL Student Research Workshop, 88–93. https://doi.org/10.18653/v1/N16-2013 ↗
  38. Zhang, Z., Robinson, D., & Tepper, J. (2018). Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network. In A. Gangemi, R. Navigli, M.-E. Vidal, P. Hitzler, R. Troncy, L. Hollink, A. Tordai, & M. Alam (Eds.), The Semantic Web (pp. 745–760). Springer International Publishing. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alshaalan-2020-hate-speech-detection-saudi-twittersphere/