Elamur Research⌕
☰
home / papers / 2024 / So Hateful! Building a Multi-Label Hate Speech Annotated Arabic Dataset

So Hateful! Building a Multi-Label Hate Speech Annotated Arabic Dataset

Paper · 2024 · Wajdi Zaghouani, Hamdy Mubarak and Md. Rafiul Biswas

Summary

The paper addresses the lack of large, richly labelled Arabic resources for detecting hate speech on social media. Arabic tweets are hard to process because writers mix dialects, spell inconsistently, and often hide hostility behind sarcasm. The authors collected a large stream of Arabic tweets from mid-2020 and filtered it by length and duplication. Because harmful posts are rare, they used existing ASAD classifiers to oversample likely offensive, hateful, adult, and spam content. This produced 15,965 tweets. Annotators labelled each tweet on many dimensions in one pass: offensiveness, hate speech and its target, emotion type and strength, effect on readers, humour versus sarcasm, factuality, importance, and spam. Roughly 15% of the tweets turned out to be offensive and 6% hateful, and the hate mostly targeted groups defined by shared ideology or politics. In baseline experiments, AraBERT beat classical classifiers trained on bag-of-words counts. The resource, SoHateful 1.0, is described as the largest fine-grained multi-label Arabic hate speech dataset. It supports work on moderation, emotion, and factuality.

Key points

  • Collection: about 60M Arabic tweets (12 Aug – 4 Oct 2020), from which a 1% random sample (600K) was drawn. Duplicates and near-duplicates were removed, as were tweets under 5 or over 80 Arabic words. Abusive tweets are cited as only 1–2% of Arabic Twitter.
  • Enrichment via ASAD:
    • Offensiveness predictions were sampled by confidence band (4,000 high, 4,000 medium, 2,000 low).
    • Sentiment classes were also sampled.
    • All predicted hate (11,000), adult (2,600), and spam (210) tweets were added.
    • After merging the overlapping pools, 15,965 tweets remained.
  • Annotation: the guidelines cover 13 questions, summarised as nine labels per tweet. Sixteen male and female annotators from several Arabic-speaking countries took part, under an experienced annotation manager.
  • Findings: 15% of tweets are offensive and 6% are hate speech, mostly aimed at ideological or political groups. Sarcasm is more frequent than humour. About 10% of tweets contain verifiable factual claims, and 7% were judged important.
  • Experiments: the data are heavily imbalanced (82.5% "No" for offensive, 93.1% "No" for hate). The authors used a 60/40 split and compared LR, SVM, RF, DT, and Gradient Boosting on CountVectorizer features against AraBERT. For the hate class, AraBERT reached about 57% precision and 66% recall, while LR's recall was only 14%. The dataset is available on request under CC BY 4.0, and the code is on GitHub.

Interesting ideas

  • I1. Use an existing weak classifier's confidence scores to stratify sampling (high, medium, low bands), so a rare phenomenon is enriched without discarding borderline or negative cases.
  • I2. Annotate each item on many orthogonal labels in one pass (offence, hate, emotion, sarcasm, factuality, spam), so a single annotation effort feeds several tasks and allows cross-label analysis.
  • I3. Keep the emotion expressed in a text separate from the effect it has on readers, instead of merging the two into one sentiment label.
  • I4. Separate generally offensive language from hate aimed at a target, and record whether the target is an individual, an organisation, or a group.
  • I5. Report per-class precision and recall instead of overall accuracy on imbalanced data, because accuracy hides near-zero recall on the minority class.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when building silver or gold sets for rare constructions, sample across witness-parser agreement levels rather than only confident cases, so evaluation keeps hard and negative examples.
  • I2 (learn): teacher-tier annotation of a sentence could record roles, speech act, irony, and factuality together, letting one labelled pool serve several stages and their consistency checks.
  • I3 (frames): pragmatics could store the speaker's expressed attitude apart from the predicted reader impact, which makes irony and implicature explanations clearer.
  • I4 (frames): an evaluative speech act could carry an explicit target slot tied to an entity or group, rather than a bare polarity flag.
  • I5 (learn): evaluations of rare phenomena (e.g., irony or rare relations) should show minority-class recall next to negative controls, so a majority-class baseline cannot look strong.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksBias, fairness & safety · 517 works

Cite

Cite as Wajdi Zaghouani, Hamdy Mubarak and Md. Rafiul Biswas. 2024. So Hateful! Building a Multi-Label Hate Speech Annotated Arabic Dataset. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zaghouani2024so,
  title = {{So Hateful! Building a Multi-Label Hate Speech Annotated Arabic Dataset}},
  author = {Wajdi Zaghouani, Hamdy Mubarak and Md. Rafiul Biswas},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/5cunk7c3ke5i},
  url = {https://doi.org/10.63317/5cunk7c3ke5i}
}

References

38 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. 2020. Daict: A dialectal arabic irony corpus extracted from twitter. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6265–6271.
  2. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16.
  3. Arfan Ahmed, Nashva Ali, Mahmood Alzubaidi, Wajdi Zaghouani, Alaa A Abd-alrazaq, and Mowafa Househ. 2022. Freely available arabic corpora: A scoping review. Computer Methods and Programs in Biomedicine Update, 2:100049. ↗
  4. Areej Al-Hassan and Hmood Al-Dossari. 2019. Detection of hate speech in social networks: a survey on multilingual corpus. In 6th international conference on computer science and information technology, volume 10, pages 10–5121. find
  5. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2018. Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. IEEE. find
  6. Ibrahim Aljarah, Maria Habib, Neveen Hijazi, Hossam Faris, Raneem Qaddoura, Bassam Hammo, Mohammad Abushariah, and Mohammad Alfawareh. 2021. Intelligent detection of hate speech in arabic social network: A machine learning approach. Journal of Information Science, 47(4):483–501. ↗
  7. Malik Almaliki, Abdulqader M Almars, Ibrahim Gad, and El-Sayed Atlam. 2023. Abmm: Arabic bert-mini model for hate-speech detection on social media. Electronics, 12(4):1048. ↗
  8. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  9. Arunkumar Bagavathi, Pedram Bashiri, Shannon Reid, Matthew Phillips, and Siddharth Krishnan. 2019. Examining untempered social media: analyzing cascades of polarized conversations. In Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, pages 625–632. ↗
  10. Mohamed Aziz Bennessir, Malek Rhouma, Hatem Haddad, and Chayma Fourati. 2022. icompass at arabic hate speech 2022: Detect hate speech using qrnn and transformers. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 176–180.
  11. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find
  12. Anis Charfi, Wajdi Zaghouani, Syed Hassan Mehdi, and Esraa Mohamed. 2019. A fine-grained annotated multi-dialectal arabic corpus. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 198–204.
  13. Despoina Chatzakou, Nicolas Kourtellis, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, and Athena Vakali. 2017. Mean birds: Detecting aggression and bullying on twitter. In Proceedings of the 2017 ACM on web science conference, pages 13–22. ↗
  14. Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, et al. 2018. Unified guidelines and resources for arabic dialect orthography. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  15. Shatha Ali A Hakami, Robert J Hendley, and Phillip Smith. 2022. A context-free arabic emoji sentiment lexicon (cf-arab-esl). In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 51–59.
  16. Sabit Hassan, Hamdy Mubarak, Ahmed Abdelali, and Kareem Darwish. 2021. Asad: Arabic social media analytics and understanding. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 113– 118.
  17. Sanaa Kaddoura, Suja A Alex, Maher Itani, Safaa Henno, Asma AlNashash, and D Jude Hemanth. 2023. Arabic spam tweets classification using deep learning. Neural Computing and Applications, pages 1–14. ↗
  18. Natasha Lomas. 2017. Facebook, google, twitter commit to hate speech action in germany. Last accessed: July. find
  19. Sarah Masud, Pinkesh Pinkesh, Amitava Das, Manish Gupta, Preslav Nakov, and Tanmoy Chakraborty. 2022. Half-day tutorial on combating online hate speech: The role of content, networks, psychology, user behavior, etc. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 1629–1631. find
  20. Binny Mathew, Ritam Dutt, Pawan Goyal, and Animesh Mukherjee. 2019. Spread of hate speech in online social media. In Proceedings of the 10th ACM conference on web science, pages 173–182. find
  21. Hanane Mohaouchane, Asmaa Mourhir, and Nikola S Nikolov. 2019. Detecting offensive language on arabic social media using deep learning. In 2019 sixth international conference on social networks analysis, management and security (SNAMS), pages 466–471. IEEE. find
  22. Hamdy Mubarak, Kareem Darwish, and Walid Magdy. 2017. Abusive language detection on arabic social media. In Proceedings of the first workshop on abusive language online, pages 52–56.
  23. Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2023. Emojis as anchors to detect arabic offensive language and hate speech. Natural Language Engineering, 29(6):1436– 1457. ↗
  24. Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, and Halima Alshabani. 2019. L-hsab: A levantine twitter dataset for hate speech and abusive language. In Proceedings of the third workshop on abusive language online, pages 111–118.
  25. Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit-Yan Yeung. 2019. Multilingual and multi-aspect hate speech analysis. arXiv preprint arXiv:1908.11049. ↗
  26. Juan Carlos Pereira-Kohatsu, Lara Quijano-Sánchez, Federico Liberatore, and Miguel Camacho-Collados. 2019. Detecting and monitoring hate speech in twitter. Sensors, 19(21):4654. ↗
  27. Jing Qian, Mai ElSherief, Elizabeth Belding, and William Yang Wang. 2018. Hierarchical cvae for fine-grained hate speech classification. arXiv preprint arXiv:1809.00088. ↗
  28. Francisco Rangel, Paolo Rosso, Wajdi Zaghouani, and Anis Charfi. 2020. Fine-grained analysis of language varieties and demographics. Natural Language Engineering, 26(6):641–661. ↗
  29. Paolo Rosso, Francisco Rangel, Irazu Hernández Farías, Leticia Cagnina, Wajdi Zaghouani, and Anis Charfi. 2018. A survey on author profiling, deception, and irony detection for the arabic language. Language and Linguistics Compass, 12(4):e12275. ↗
  30. Ahmad Shapiro, Ayman Khalafallah, and Marwan Torki. 2022. Alexu-aic at arabic hate speech 2022: Contrast to classify. arXiv preprint arXiv:2207.08557. ↗
  31. Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. arXiv preprint arXiv:1705.09899. ↗
  32. Zeerak Waseem and Dirk Hovy. 2016. Understanding abuse: A typology of abusive language detection subtasks.? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88–93.
  33. Hajime Watanabe, Mondher Bouazizi, and Tomoaki Ohtsuki. 2018. Hate speech on twitter: A pragmatic approach to collect hateful and offensive expressions and perform hate speech detection. IEEE access, 6:13825–13835. ↗
  34. Wajdi Zaghouani. 2014. Critical survey of the freely available arabic corpora. In International Conference on Language Resources and Evaluation (LREC’2014), OSACT Workshop. Reykjavik, Iceland, 26-31 May 2014. ↗
  35. Wajdi Zaghouani, Houda Bouamor, Abdelati Hawwari, Mona Diab, Ossama Obeid, Mahmoud Ghoneim, Sawsan Alqahtani, and Kemal Oflazer. 2016a. Guidelines and framework for a large scale arabic diacritized corpus. In The Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 3637–3643. European Language Resources Association (ELRA).
  36. Wajdi Zaghouani and Anis Charfi. 2018. Guidelines and annotation framework for arabic author profiling. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). ↗
  37. Wajdi Zaghouani, Nizar Habash, Ossama Obeid, Behrang Mohit, Houda Bouamor, and Kemal Oflazer. 2016b. Building an arabic machine translation post-edited corpus: Guidelines and annotation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1869–1876.
  38. Wajdi Zaghouani, Behrang Mohit, Nizar Habash, Ossama Obeid, Nadi Tomeh, Alla Rozovskaya, Noura Farra, Sarah Alkuhlani, and Kemal Oflazer. 2014. Large scale arabic error annotation: Guidelines and framework. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaghouani-2024-so-hateful-building-multi-label/