Elamur Research⌕
☰
home / papers / 2024 / The Annotators Agree to Not Agree on the Fine-grained Annotation of Hate-speech against Women in Algerian Dialect Comments

The Annotators Agree to Not Agree on the Fine-grained Annotation of Hate-speech against Women in Algerian Dialect Comments

Paper · 2024 · Imane Guellil, Yousra Houichi, Sara Chennoufi et al.

Summary

The paper addresses the shortage of resources for detecting sexist hate speech in Arabic, and in Algerian dialect in particular. Earlier Arabic misogyny datasets mostly used two or three labels. The authors built a YouTube corpus with much finer-grained annotation. They picked 335 videos about women using feminine adjectives as search keywords, then harvested 373,984 comments and replies through the YouTube Data API. From these, 10,000 comments were randomly sampled. The sample mixes MSA and Algerian dialect in Arabic script and Arabizi, with some French and English. Three native-speaker annotators worked independently over three weeks with a shared guideline. They used 13 classes: insult, vulgarity, and hate, plus their combinations, along with labels for off-topic, positive, emoji-only, Quran passages and undecidable items. Agreement turned out low, with Kappa around 35%. The authors present this disagreement as a central finding: richer label sets increase subjectivity and the risk of bias. They also argue that guidelines need to be clearer. The resource is described as work in progress and is promised to be freely released.

Key points

  • The corpus is named Corpus_Youtube_women_10000. It contains 10,000 annotated comments sampled from 373,984 comments collected from 335 manually vetted video_ids.
  • There are 13 classes: 0, i, v, h, s, b, p, e, c, iv, ih, vh, ivh. The conclusion mentions 14 labels, which is inconsistent with the rest of the paper.
  • Annotation was done by three native speakers (two women, one man). They worked separately over three weeks and followed a written guideline.
  • Inter-annotator agreement was low, with Kappa of roughly 35%. Annotators disagreed more as the number of classes grew.
  • The work extends the authors' earlier binary corpus of 5,000 YouTube comments (Ara-women-hate, Guellil et al. 2022).

Interesting ideas

  • I1. Overlapping categories are encoded as a single compound label with a fixed letter order (i, then v, then h, e.g. "vh" and never "hv"). This keeps multi-label annotation simple while still letting it be counted and compared.
  • I2. A dedicated "problem" label lets annotators mark items they cannot decide, instead of forcing a guess. They are told to use it sparingly.
  • I3. Non-content items get their own classes: off-topic comments, emoji-only comments, and quoted religious text. This separates these items from the real target categories, so they are not miscounted as negatives.
  • I4. Low agreement is reported as a result in itself. The authors link it to label granularity and annotator subjectivity rather than hiding it.
  • I5. Data is gathered in two stages: keyword search first, then manual vetting of the sources (videos) before bulk automatic extraction.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Speech-act or irony tags that can co-occur could get a canonical compound form, so that comparing outputs across witnesses never fails on ordering noise.
  • I2 (learn): When LLM teacher tiers or witness parsers label silver data, an explicit "undecidable" output would let ambiguous sentences be routed aside rather than polluting training.
  • I3 (answer): Reader components could first classify inputs as out-of-scope or quotation-only before attempting comprehension, which gives cleaner negative controls in evaluation.
  • I4 (learn): Evaluation could track how witness agreement drops as tag sets get finer, and use that curve to decide which distinctions are reliable enough to train on.
  • I5 (world): When compiling text sources for knowledge or fable QA, vetting at the source level before mass harvesting cheaply raises topical relevance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksBias, fairness & safety · 517 works

Cite

Cite as Imane Guellil, Yousra Houichi, Sara Chennoufi, Mohamed Boubred, Anfal Yousra Boucetta and Faical Azouaou. 2024. The Annotators Agree to Not Agree on the Fine-grained Annotation of Hate-speech against Women in Algerian Dialect Comments. Proceedings of the Fifth Workshop on Resources for African Indigenous Languages @ LREC-COLING 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{guellil2024annotators,
  title = {{The Annotators Agree to Not Agree on the Fine-grained Annotation of Hate-speech against Women in Algerian Dialect Comments}},
  author = {Imane Guellil, Yousra Houichi, Sara Chennoufi, Mohamed Boubred, Anfal Yousra Boucetta and Faical Azouaou},
  booktitle = {Proceedings of the Fifth Workshop on Resources for African Indigenous Languages @ LREC-COLING 2024},
  year = {2024},
  doi = {10.63317/2km3xkkar9nt},
  url = {https://doi.org/10.63317/2km3xkkar9nt}
}

References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Areej Al-Hassan and Hmood Al-Dossari. 2019. Detection of hate speech in social networks: A survey on multilingual corpus. Computer Science & Information Technology (CS & IT), 9(2):83. find
  2. Albadi, Nuha and Kurdi, Maram and Mishra, Shivakant. 2018. Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. IEEE. Dina Almanea and Massimo Poesio. Armis-the arabic misogyny and sexism corpus with annotator subjective disagreements. find
  3. Sattam Almatarneh, Pablo Gamallo, Francisco J Ribadas Pena, and Alexey Alexeev. 2019. Supervised classifiers to identify hate speech on english and spanish tweets. In International Conference on Asian Digital Libraries, pages 23–30. Springer. find
  4. Safa Alsafari, Samira Sadaoui, and Malek Mouhoub. 2020. Hate and offensive speech detection on arabic social media. Online Social Networks and Media, 19:100096. ↗
  5. Oussama Boucherit and Kheireddine Abainia. 2022. Offensive language detection in under-resourced algerian dialectal arabic language. arXiv preprint arXiv:2203.10024. ↗
  6. Manuela Caiani, Benedetta Carlotti, and Enrico Padoan. 2021. Online hate speech and the radical right in times of pandemic: The italian and english cases. Javnost-The Public, 28(2):202– 218. Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Abdelali, Soon-gyo Jung, Bernard J https://efi.ed.ac.uk/ ↗
  7. Jansen, and Joni Salminen. 2020. A multi-platform arabic news comment dataset for offensive language detection. In Proceedings of the 12th language resources and evaluation conference, pages 6203–6212.
  8. Angel Felipe Magnossão de Paula, Roberto Fray da Silva, and Ipek Baris Schlicht. 2021. Sexism prediction in spanish and english tweets using monolingual and multilingual bert and ensemble models. arXiv preprint arXiv:2111.04551. ↗
  9. Zineb El Abboubi, Ahmadou Bouylmani, and Mohammed Derdar. 2020. Sexism in moroccan arabic: Gender differences in perceptions and use of language. Journal of Applied Language and Culture Studies, 3:215–230. find
  10. Karima Benatchba Prof President ESI. 2016. A Sentiment analysis approach for Arabic dialects texts analysis based on automatic translation: Application to the Algerian dialect. Ph.D. thesis, ESI Algeria. find
  11. Imane Guellil, Ahsan Adeel, Faical Azouaou, Mohamed Boubred, Yousra Houichi, and Akram Abdelhaq Moumna. 2022. Ara-women-hate: An annotated corpus dedicated to hate speech detection against women in the arabic community. page 68–75.
  12. Imane Guellil, Faical Azouaou, Fodil Benali, Ala Eddine Hachani, and Marcelo Mendoza. 2020a. The role of transliteration in the process of arabizi translation/sentiment analysis. Recent Advances in NLP: The Case of Arabic Language, pages 101–128. ↗
  13. Imane Guellil, Faical Azouaou, Fodil Benali, alaeddine Hachani, and Houda Saadane. 2018. Approche hybride pour la translitération de l’arabizi algérien : une étude préliminaire. In Conference: 25e conférence sur le Traitement Automatique des Langues Naturelles (TALN), May 2018, Rennes, FranceAt: Rennes, France. https://www.researchgate.net/publication . . . . ↗
  14. Imane Guellil, Faical Azouaou, and Francisco Chiclana. 2020b. Arautosenti: automatic annotation and new tendencies for sentiment classification of arabic messages. Social Network Analysis and Mining, 10:1–20. ↗
  15. Imane Guellil and Azouaou Faical. 2017. Bilingual lexicon for algerian arabic dialect treatment in social media. In WiNLP: Women & Underrepresented Minorities in Natural Language Processing (co-located with ACL 2017). find
  16. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. 2021. Arabic natural language processing: An overview. Journal of King Saud University-Computer and Information Sciences, 33(5):497–507.
  17. Bushr Haddad, Zoher Orabe, Anas Al-Abood, and Nada Ghneim. 2020. Arabic offensive language detection with attention-based deep neural networks. In Proceedings of the 4th workshop on open-source Arabic corpora and processing tools, with a shared task on offensive language detection, pages 76–81. ↗
  18. Fatemah Husain. 2020. Arabic offensive language detection using machine learning and ensemble machine learning approaches. arXiv preprint arXiv:2005.08946. ↗
  19. Adaikkan Kalaivani and Durairaj Thenmozhi. 2021. Multilingual hate speech and offensive language detection in english, hindi, and marathi languages. find
  20. Rohan Kshirsagar, Tyus Cukuvac, Kathleen McKeown, and Susan McGregor. 2018. Predictive embeddings for hate speech detection on twitter. arXiv preprint arXiv:1809.10644. ↗
  21. Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2022. Emojis as anchors to detect arabic offensive language and hate speech. arXiv preprint arXiv:2201.06723. ↗
  22. Hamdy Mubarak, Ammar Rashed, Kareem Darwish, Younes Samih, and Ahmed Abdelali. 2020. Arabic offensive language on twitter: Analysis and experiments. arXiv preprint arXiv:2004.02192. ↗
  23. Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, and Halima Alshabani. 2019. L-hsab: A levantine twitter dataset for hate speech and abusive language. In Proceedings of the third workshop on abusive language online, pages 111–118.
  24. Endang Wahyu Pamungkas, Alessandra Teresa Cignarella, Valerio Basile, Viviana Patti, et al. 2018. Automatic identification of misogyny in english and italian tweets at evalita 2018 with a multilingual hate lexicon. In Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (EVALITA 2018), volume 2263, pages 1–6. CEUR-WS. ↗
  25. Georgios K Pitsilis, Heri Ramampiaro, and Helge Langseth. 2018. Detecting offensive language in tweets using deep learning. arXiv preprint arXiv:1801.04433. ↗
  26. Neil Salkind. 2010. Cohen’s kappa. pages 188– 189. find
  27. Fatima Shannaq, Bassam Hammo, Hossam Faris, and Pedro A Castillo-Valdivieso. 2022. Offensive language detection in arabic social networks using evolutionary-based classifiers learned from fine-tuned embeddings. IEEE Access, 10:75018–75039. ↗
  28. Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88–93.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guellil-2024-annotators-agree-not-agree-fine/