Elamur Research⌕
☰
home / papers / 2025 / An Annotated Corpus of Arabic Tweets for Hate Speech Analysis

An Annotated Corpus of Arabic Tweets for Hate Speech Analysis

Paper · 2025 · Wajdi Zaghouani and Md. Rafiul Biswas

Summary

The paper targets a shortage of labeled Arabic data for detecting hateful content. Arabic social media mixes many informal dialects, and most existing resources treat hate speech as a single-label problem. The authors drew on a pool of about 60 million tweets from mid-2020. They filtered the pool and used the ASAD toolkit to pick candidates across several confidence and sentiment strata, which yielded 10,000 tweets. Five annotators from four Arab countries, supervised by an adjudicating manager, labeled each tweet as offensive or clean. Offensive tweets also received one or more target labels, such as politics, religion, origin or ethnicity, gender, and others. Agreement was reported as high for the offensive/clean decision and somewhat lower for targets. Fine-tuned transformers served as a check on the annotations. AraBERTv2 performed best on the binary task and on target classification, ahead of CAMeLBERT and XLM-RoBERTa. The resource is the first Arabic tweet corpus in which every applicable hate target is marked. It is released under CC BY 4.0, with access granted on request.

Key points

  • Data collection and sampling
    • Source: roughly 60M Arabic tweets from 12 August to 4 October 2020, of which 600,000 (1%) were randomly sampled.
    • Filtering removed duplicates, incomplete tweets, and tweets under 5 or over 80 words.
    • The final 10,000 tweets came from ASAD-based selection by offensive-language confidence (high, average, low bands), by sentiment (positive, negative, neutral), and by hate, adult and spam flags.
  • Annotation setup
    • Five annotators: two from Tunisia, one each from Egypt, Jordan and Qatar (three women, two men).
    • They worked in MicroMappers and met weekly.
    • A manager resolved conflicts, analyzed errors and revised the guidelines.
  • Agreement
    • The abstract reports 0.86 for offensiveness and 0.71 for multi-target labels.
    • The body reports a Fleiss' Kappa of 0.8143 for offensive vs. not.
  • Label distribution
    • 6,036 tweets were labeled offensive.
    • Target shares: political ideology/sports 28.76%, Other 28.51%, origin/ethnicity/country 26.20%, religion/sect 13.51%, social class/profession 2.47%.
    • Physical appearance (0.25%) and gender (0.30%) were rare.
  • Models
    • Setup: 70/20/10 train/test/validation split, batch 4, 3 epochs, learning rate 2e-5, sigmoid-based loss, Tesla T4 GPU.
    • Offensive vs. clean: AraBERTv2 reached F1 0.7865 and accuracy 0.786; CAMeLBERT reached 0.736.
    • Target classification: AraBERTv2 reached F1 0.6889, precision 0.7172 and recall 0.6628.

Interesting ideas

  • I1. Selecting items for annotation by stratifying an existing classifier's confidence bands (high, middle, low) yields a pool that mixes clear cases with borderline ones, rather than just easy positives.
  • I2. Overlapping selection streams (by confidence, sentiment and content flags) are merged and deduplicated, so one pool covers several phenomena at once.
  • I3. Annotators were chosen deliberately to span dialect regions, because the same phrase can be read differently across cultures.
  • I4. A single manager adjudicates conflicts, studies the errors and keeps updating the guidelines, so the annotation scheme evolves with the data.
  • I5. Hateful content is labeled with every applicable target instead of one dominant class.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When choosing sentences to send to an LLM teacher or human review, sample across witness-parser agreement bands so hard and contested parses are well represented.
  • I2 (learn): Silver-data batches could be built by merging streams keyed on different weaknesses (rare relations, long sentences, coordination), then deduplicating.
  • I3 (frames): Irony and implicature labels could be collected from readers of different English varieties, keeping disagreement as a signal rather than averaging it away.
  • I4 (learn): Self-induced repair rules could follow the same loop: cluster disagreements, revise one explicit rule, and re-check agreement after each revision.
  • I5 (frames): Speech-act and frame annotation could allow several simultaneous labels per utterance (e.g., insult plus assertion), making the frame layer more faithful and easier to explain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksSemitic languages · 3,066 worksBias, fairness & safety · 587 works

Cite

Cite as Wajdi Zaghouani and Md. Rafiul Biswas. 2025. An Annotated Corpus of Arabic Tweets for Hate Speech Analysis. Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI Era.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zaghouani2025annotated,
  title = {{An Annotated Corpus of Arabic Tweets for Hate Speech Analysis}},
  author = {Wajdi Zaghouani and Md. Rafiul Biswas},
  booktitle = {Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI Era},
  year = {2025},
  url = {https://aclanthology.org/2025.ranlp-1.163/}
}

References

22 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mahmoud Mohamed Abdelsamie, Shahira Shaaban Azab, and Hesham A Hefny. 2024. A comprehensive review on arabic offensive language and hate speech detection on social media: methods, challenges and solutions. Social Network Analysis and Mining, 14(1):1–49. • Dataset Size: Our dataset contains only 10000 tweets which is limited for generability Razan Abdul-Nabi, Rasha Obeidat, and Anas Bsoul. 2024. A survey on machine translation of low-resource arabic dialects. In 2024 15th International Limitation and Future Scope Conference on Information and Communication Systems (ICICS), pages 1–6. IEEE. Association for Computational Linguistics: System Demonstrations, pages 113–118. ↗
  2. Rogayah M Al-Ibrahim, Mostafa Z Ali, and Hassan M Najadat. 2023. Detection of hateful social media content for arabic language. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(9):1–26. ↗
  3. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  4. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2018. Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In Proceedings of the 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, pages 69–76. ACM. find
  5. Ali Alhazmi, Rohana Mahmud, Norisma Idris, Mohamed Elhag Mohamed Abo, and Christopher Eke. 2024a. A systematic literature review of hate speech identification on arabic twitter data: research challenges and future directions. PeerJ Computer Science, 10:e1966. ↗
  6. Ali Alhazmi, Rohana Mahmud, Norisma Idris, Mohamed Elhag Mohamed Abo, and Christopher Ifeanyi Eke. 2024b. Code-mixing unveiled: Enhancing the hate speech detection in arabic dialect tweets using machine learning models. Plos one, 19(7):e0305657. ↗
  7. Fatimah Alkomah and Xiaogang Ma. 2022. A literature review of textual hate speech detection methods and datasets. Information, 13(6):273. ↗
  8. Safa Alsafari, Samira Sadaoui, and Malek Mouhoub. 2020. Hate and offensive speech detection on arabic social media. Online Social Networks and Media, 19:100096. ↗
  9. Raghad Alshaalan and Hend Al-Khalifa. 2020. Hate speech detection in saudi twittersphere: A deep learning approach. In Proceedings of the fifth Arabic natural language processing workshop, pages 12–23.
  10. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  11. A Conneau. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  12. Matthew Costello and James Hawdon. 2020. Hate speech in online spaces. The Palgrave handbook of international cybercrime and cyberdeviance, pages 1397–1416. ↗
  13. Bharat Gaind, Varun Syal, and Sneha Padgalwar. 2019. Emotion detection and analysis on social media. arXiv preprint arXiv:1901.08458. ↗
  14. Sabit Hassan, Hamdy Mubarak, Ahmed Abdelali, and Kareem Darwish. 2021. Asad: Arabic social media analytics and understanding. In Proceedings of the 16th Conference of the European Chapter of the V Kakulapati, Sheri Mahender Reddy, and A Paramasivam. 2023. Managing postpandemic effects using artificial intelligence with human-computer interaction. In Innovations in Artificial Intelligence and Human-Computer Interaction in the Digital Era, pages 207– 232. Elsevier.
  15. Julia Kansok-Dusche, Cindy Ballaschk, Norman Krause, Anke Zeißig, Lisanne Seemann-Herz, Sebastian Wachs, and Ludwig Bilz. 2023. A systematic review on hate speech among children and adolescents: Definitions, prevalence, and overlap with related phenomena. Trauma, violence, & abuse, 24(4):2598– 2615. ↗
  16. Ramzi Khezzar, Abdelrahman Moursi, and Zaher Al Aghbari. 2023. arhatedetector: detection of hate speech from standard and dialectal arabic tweets. Discover Internet of Things, 3(1):1. ↗
  17. Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, and Halima Alshabani. 2019. L-hsab: A levantine twitter dataset for hate speech and abusive language. In Proceedings of the third workshop on abusive language online, pages 111–118.
  18. Juan Carlos Pereira-Kohatsu, Lara Quijano-Sánchez, Federico Liberatore, and Miguel Camacho-Collados. 2019. Detecting and monitoring hate speech in twitter. Sensors, 19(21):4654. ↗
  19. Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet B Pierrehumbert. 2021. Two contrasting data annotation paradigms for subjective nlp tasks. arXiv preprint arXiv:2112.07475. ↗
  20. Cheng Xian-Yi, Zhu Ling-ling, Zhu Qian, and Wang Jin. 2010. The framework of network public opinion monitoring and analyzing system based on semantic content identification. Journal of Convergence Information Technology, 5(10):1–5. find
  21. Wajdi Zaghouani, Behrang Mohit, Nizar Habash, Ossama Obeid, Nadi Tomeh, Alla Rozovskaya, Noura Farra, Sarah Alkuhlani, and Kemal Oflazer. 2014. Large scale arabic error annotation: Guidelines and framework. ↗
  22. Wajdi Zaghouani, Hamdy Mubarak, and Md Rafiul Biswas. 2024. So hateful! building a multi-label hate speech annotated arabic dataset. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 15044–15055.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaghouani-2025-annotated-corpus-arabic-tweets-hate/