Elamur Research⌕
☰
home / papers / 2024 / AraTar: A Corpus to Support the Fine-grained Detection of Hate Speech Targets in the Arabic Language

AraTar: A Corpus to Support the Fine-grained Detection of Hate Speech Targets in the Arabic Language

Summary

Most Arabic hate speech resources only separate hateful from non-hateful text, or assign a single hate type per message. They rarely say who is being attacked. The authors address two tasks. Task 1 is multi-label detection of the hate type. Task 2 is multi-label identification of the specific target group inside each type.

They pooled hateful tweets from five public corpora, including Arabic-Twitter, OSACT5 and the Arabic hate-speech corpus. They then re-annotated the tweets under one hierarchical scheme, producing AraTar. The authors describe it as the first and largest Arabic corpus labelled with both hate types and targets in a multi-label setting.

Three model families were compared on AraTar:

  • an SVM with binary relevance,
  • an LSTM fed with contextual embeddings,
  • fine-tuned language models (MARBERTv2 and AraBERT-twitter, base and large).

Fine-tuned AraBERTv0.2-twitter (base) performed best on both tasks. AraTar matters because it offers a common labelling standard where earlier Arabic datasets each used incompatible label sets.

Key points

  • Task 1 labels: religion-hate (RH), ethnicity-hate (EH), nationality-hate (NH), gender-hate (GH), undefined-hate (UDH) and clean (CL).

  • Task 2 labels: targets nested under each type, for example:

    • Islam, Judaism and Christianity under RH;
    • Arab and African under EH;
    • Iranian, Israeli, Saudi, Turkish, Qatari and American under NH;
    • Females and Males under GH;
    • an "undefined" target under each type.

  • Annotation process: three native-speaker volunteers worked over six months, after a training workshop and a 300-tweet pilot.

    • Stage 1: 1,541 tweets (30%) were triple-annotated so inter-annotator agreement could be measured.
    • Stage 2: the remaining 3,594 tweets were split for single annotation, 1,198 per annotator.

  • Best results: AraBERTv0.2-twitter (base) reached the following scores.

    TaskMicro-F1Macro-F1
    Task 184.5%77.46%
    Task 285.03%73.15%

  • Weak spots: UDH and rare targets were hard for every model, and some undefined-target classes scored zero.

  • Cross-corpus evaluation: the same model was also tested on Arabic-Twitter and OSACT5. On OSACT5, several minority classes collapsed to zero F1.

Interesting ideas

  • I1. A two-level label hierarchy (type first, then target within that type), with an explicit "undefined" bucket under every parent, captures vague cases without forcing a wrong fine label. — For Mova (frames): Speech-act and irony labels could gain a per-parent "unspecified" slot, and the attacked or addressed party could be stored as an explicit target role rather than left implicit.
  • I2. Pooling several existing datasets and relabelling them under one shared scheme fixes the problem of each source using its own incompatible categories. — For Mova (tree): The same approach suits merging treebanks with diverging conventions, by converting all of them to one UD dialect before training gparse or the perceptron parser.
  • I3. The pilot compared each annotator to a lead annotator's labels to find the most consistent person and to revise the guidelines. Afterwards, only a 30% slice was annotated in triplicate and the rest singly, which saves effort while still measuring reliability. — For Mova (learn): Silver data could follow this pattern: run all witness parsers on a calibration slice to rank their reliability, then rely on cheaper single-witness labels elsewhere.
  • I4. Multi-label framing, instead of mutually exclusive classes, acknowledges that one message can attack several groups at once. — For Mova (world): A single sentence may change the states of several entities, so world updates should allow multiple simultaneous effects rather than picking one.
  • I5. Per-class reporting and testing on outside corpora exposed classes that silently scored zero despite good micro averages. — For Mova (learn): Evaluation should always break scores down per label and per source corpus, so that collapsed rare categories show up rather than hiding behind aggregate metrics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Sentiment & emotion · Semitic languages · Bias, fairness & safety

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alghamdi2024aratar,
  title = {{AraTar: A Corpus to Support the Fine-grained Detection of Hate Speech Targets in the Arabic Language}},
  author = {Alghamdi, Seham and Benkhedda, Youcef and Alharbi, Basma and Batista-Navarro, Riza},
  booktitle = {Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024},
  year = {2024},
  doi = {10.63317/54jsv94d2ew9},
  url = {https://doi.org/10.63317/54jsv94d2ew9}
}

References

41 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Istabrak Abbes, Eya Nakache, and Moez BenHajHmida. 2021. Context-aware language modeling for arabic misogyny identification. In Working Notes of the 2021 Forum for Information Retrieval Evaluation (FIRE 2021), pages 847–851. find
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105. Association for Computational Linguistics. find our card
  3. Ibrahim Abu Farha and Walid Magdy. 2020. Multitask Learning for Arabic Offensive Language and Hate-Speech Detection. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 86–90, Marseille, France. European Language Resource Association. find
  4. Areej Al-Hassan and Hmood Al-Dossari. 2022. Detection of hate speech in Arabic tweets using deep learning. Multimedia systems, 28(6):1963– 1974. find
  5. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2018. Are they Our Brothers? Analysis and Detection of Religious Hate Speech in the Arabic Twittersphere. In Proceedings of the 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. find
  6. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2019. Investigating the effect of combining GRU neural networks with handcrafted features for religious hatred detection on Arabic Twitter space. Social Network Analysis and Mining, 9(1):41. find
  7. Badr AlKhamissi and Mona Diab. 2022. Meta AI at Arabic Hate Speech 2022: MultiTask Learning with Self-Correction for Hate Speech Classification. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 186– 193, Marseille, France. European Language Resources Association. find
  8. Afaf Alotaibi and Mozaherul Hoque Abul Hasanat. 2020. Racism Detection in Twitter Using Deep Learning and Text Mining Techniques for the Arabic Language. In Proceedings of the 1st International Conference of Smart Systems and Emerging Technologies (SMARTTECH), pages 161–164. find
  9. Rawan Abdullah Alraddadi and Moulay Ibrahim El-Khalil Ghembaza. 2021. Anti-Islamic Arabic Text Categorization using Text Mining and Sentiment Analysis Techniques. International Journal of Advanced Computer Science and Applications, 12(8). find
  10. Safa Alsafari and Samira Sadaoui. 2021. Ensemble-based semi-supervised learning for hate speech detection. In The International FLAIRS Conference Proceedings, volume 34. find
  11. Safa Alsafari, Samira Sadaoui, and Malek Mouhoub. 2020a. Deep Learning Ensembles for Hate Speech Detection. In Proceedings of the 32nd IEEE International Conference on Tools with Artificial Intelligence (ICTAI), pages 526– 531. find
  12. Safa Alsafari, Samira Sadaoui, and Malek Mouhoub. 2020b. Hate and offensive speech detection on Arabic social media. Online Social Networks and Media, 19:100096. find
  13. Maha Jarallah Althobaiti. 2022. BERT-based Approach to Arabic Hate Speech and Offensive Language Detection in Twitter: Exploiting Emojis and Sentiment Analysis. International Journal of Advanced Computer Science and Applications, 13(5). find
  14. Sai Saketh Aluru, Binny Mathew, Punyajoy Saha, and Animesh Mukherjee. 2020. Deep learning models for multilingual hate speech detection. arXiv preprint arXiv:2004.06465. ↗
  15. Faisal Yousif Al Anezi. 2022. Arabic Hate Speech Detection Using Deep Recurrent Neural Networks. Applied Sciences, 12(12):6010. find
  16. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based Model for Arabic Language Understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. find our card
  17. Abdullah Aref, Rana Husni Al Mahmoud, Khaled Taha, and Mahmoud Al-Sharif. 2020. Hate Speech Detection of Arabic Shorttext. In Proceedings of the 9th International Conference on Information Technology Convergence and Services (ITCSE 2020), pages 81–94. AIRCC Publishing Corporation. find
  18. Salma Azzi and Chiraz Zribi. 2022. Comparing Deep Learning Models for Multi-label Classification of Arabic Abusive Texts in Social Media. In Proceedings of the 17th International Conference on Software Technologies, pages 374–381, Lisbon, Portugal. SCITEPRESS - Science and Technology Publications. find
  19. Mohamed Aziz Bennessir, Malek Rhouma, Hatem Haddad, and Chayma Fourati. 2022. iCompass at Arabic hate speech 2022: Detect hate speech using QRNN and transformers. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 176–180, Marseille, France. European Language Resources Association. find
  20. Catherine Blaya. 2019. Cyberhate: A review and content analysis of intervention strategies. Aggression and Violent Behavior, 45:163–172. find
  21. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning, 20(3):273–297. find
  22. Rehab Duwairi, Amena Hayajneh, and Muhannad Quwaider. 2021. A Deep Learning Framework for Automatic Detection of Hate Speech Embedded in Arabic Tweets. Arabian Journal for Science and Engineering, 46(4):4001–4014. find
  23. Mai ElSherief, Vivek Kulkarni, Dana Nguyen, William Yang Wang, and Elizabeth Belding. 2018. Hate Lingo: A Target-Based Linguistic Analysis of Hate Speech in Social Media. In Proceedings of the International AAAI Conference on Web and Social Media, volume 12. find
  24. Imane Guellil, Ahsan Adeel, Faical Azouaou, Mohamed Boubred, Yousra Houichi, and Akram Abdelhaq Moumna. 2022. Ara-Women-Hate: An Annotated Corpus Dedicated to Hate Speech Detection against Women in the Arabic Community. In Proceedings of the Workshop on Dataset Creation for Lower-Resourced Languages within the 13th Language Resources and Evaluation Conference, pages 68–75, Marseille, France. European Language Resources Association. find
  25. George Hripcsak and Adam S. Rothschild. 2005. Agreement, the F-Measure, and Reliability in Information Retrieval. Journal of the American Medical Informatics Association: JAMIA, 12(3):296–298. find
  26. Klaus Krippendorff. 1970. Bivariate agreement coefficients for reliability of data. Sociological Methodology, 2:139–150. find
  27. Klaus Krippendorff. 2004. Measuring the reliability of qualitative text analysis data. Quality and quantity, 38:787–800. find
  28. Angel Felipe Magnossão de Paula, Paolo Rosso, Imene Bensalem, and Wajdi Zaghouani. 2022. UPV at the Arabic Hate Speech 2022 Shared Task: Offensive Language and Hate Speech Detection using Transformers and Ensemble Models. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 181– 185, Marseille, France. European Language Resources Association. find
  29. Abdelkader El Mahdaouy, Abdellah El Mekki, Ahmed Oumar, Hajar Mousannif, and Ismail Berrada. 2022. Deep Multi-Task Models for Misogyny Identification and Categorization on Arabic Social Media. arXiv preprint arXiv:2206.08407. ↗ our card
  30. Abir Messaoudi, Chayma Fourati, Mayssa Kchaou, and Hatem Haddad. 2021. iCompass Working Notes for Arabic Misogyny Identification. In Working Notes of the 2021 Forum for Information Retrieval Evaluation (FIRE 2021), page 5. find
  31. Djamila Mohdeb, Meriem Laifa, Fayssal Zerargui, and Omar Benzaoui. 2022. Evaluating transfer learning approach for detecting Arabic anti-refugee/migrant speech on social media. Aslib Journal of Information Management, 74(6):1070– 1088. find
  32. Hamdy Mubarak, Hend Al-Khalifa, and Abdulmohsen Al-Thubaity. 2022. Overview of OS-ACT5 shared task on Arabic offensive language and hate speech detection. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 162–166, Marseille, France. European Language Resources Association. find
  33. Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2023. Emojis as anchors to detect Arabic offensive language and hate speech. Natural Language Engineering, 29(6):1436– 1457. find our card
  34. Hala Mulki and Bilal Ghanem. 2021. ArMI at FIRE 2021: Overview of the First Shared Task on Arabic Misogyny Identification. In Working Notes of the 2021 Forum for Information Retrieval Evaluation (FIRE 2021), pages 820–830. find
  35. Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, and Halima Alshabani. 2019. L-HSAB: A Levantine Twitter Dataset for Hate Speech and Abusive Language. In Proceedings of the 3rd Workshop on Abusive Language Online, pages 111–118, Florence, Italy. Association for Computational Linguistics. find
  36. Abdusalam Nwesri, Stephen Wu, and Harmain Harmain. 2021. Detecting Misogyny in Arabic Tweets. In Working Notes of the 2021 Forum for Information Retrieval Evaluation (FIRE 2021), page 6. find
  37. Ahmad Shapiro, Ayman Khalafallah, and Marwan Torki. 2022. AlexU-AIC at Arabic Hate Speech 2022: Contrast to Classify. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 200–208, Marseille, France. European Language Resources Association. find
  38. Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11– 21. find
  39. Zeerak Waseem and Dirk Hovy. 2016. Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. In Proceedings of the NAACL Student Research Workshop, pages 88–93, San Diego, California. Association for Computational Linguistics. find our card
  40. Ankit Yadav, Shubham Chandel, Sushant Chatufale, and Anil Bandhakavi. 2023. LAHM: Large Annotated Dataset for Multi-Domain and Multilingual Hate Speech Identification. arXiv preprint arXiv:2304.00913. ↗
  41. Zhuoyi Yang, Ming Ding, Yanhui Guo, Qingsong Lv, and Jie Tang. 2022. Parameter-Efficient Tuning Makes a Good Classification Head. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7576–7586. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alghamdi-2024-aratar-corpus-support-fine-grained/