Elamur Research⌕
☰
home / papers / 2026 / Beyond Annotator Disagreement: Guideline-Induced Errors in Arabic Hate Speech Annotation

Beyond Annotator Disagreement: Guideline-Induced Errors in Arabic Hate Speech Annotation

Paper · 2026 · Wajdi Zaghouani

Summary

Errors in hate speech corpora are usually blamed on annotators: their bias, their disagreement, or weak training. This position paper places a large share of errors earlier in the pipeline, in the annotation guidelines. If a scheme ignores the linguistic and cultural realities of the discourse it covers, some mislabels will happen however careful the annotators are. The author examines Arabic social media, where four factors raise this risk: dialect variation, insults tied to lineage, tribe and sect, praise used sarcastically, and intra-religious delegitimization. Six case studies built on attested Arabic examples illustrate three failure mechanisms. The paper then offers a taxonomy of five guideline-induced error types, maps each mechanism to those types, and proposes a four-stage diagnostic procedure for dataset builders. It also separates such errors from legitimate human label variation (HLV). Disagreement that follows annotators' backgrounds rather than the text points to a guideline that needs repair, not to variation worth modeling. The work is explicitly theoretical. It matters because it treats annotation quality as a design problem to fix before labeling begins.

Key points

  • The three mechanisms are:
    • cultural misclassification: hostile expressions specific to the culture have no matching category;
    • dialectal ambiguity: a word can be offensive in one variety (Gulf, Levantine, Egyptian, Maghrebi) and harmless or affectionate in another;
    • annotation projection: English-oriented moderation categories are reused without adaptation.
  • The contributions are six illustrative case studies, a five-type error taxonomy, a mechanism-to-error mapping, and a four-stage diagnostic framework.
  • Existing Arabic datasets discussed include those of Mubarak et al. (2017) and Albadi et al. (2018), plus L-HSAB (Mulki et al., 2019). The paper cites Haj Ahmed et al. (2025), who found a strong Lebanese-dialect bias in L-HSAB.
  • There is no quantitative evaluation yet. The author suggests a pilot on L-HSAB that would measure IAA and dialect-conditioned agreement before and after applying the framework.
  • No datasets or annotation resources are released. The work was funded by QRDI grant NPRP14C-0916-210015.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northwestern University in QatarDoha · Qatar39 works · 148 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksSemitic languages · 3,066 worksBias, fairness & safety · 587 works

Cite

Cite as Wajdi Zaghouani. 2026. Beyond Annotator Disagreement: Guideline-Induced Errors in Arabic Hate Speech Annotation. Proceedings of the 20th Linguistic Annotation Workshop (LAW XX).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.law-main.5

@inproceedings{zaghouani2026beyond,
  title = {{Beyond Annotator Disagreement: Guideline-Induced Errors in Arabic Hate Speech Annotation}},
  author = {Wajdi Zaghouani},
  booktitle = {Proceedings of the 20th Linguistic Annotation Workshop (LAW XX)},
  year = {2026},
  doi = {10.18653/v1/2026.law-main.5},
  url = {https://doi.org/10.18653/v1/2026.law-main.5}
}

References

27 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha and Walid Magdy. 2020. From Arabic sentiment analysis to sarcasm detection: The ArSarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39, Marseille, France. European Language Resource Association.
  2. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2018. Are they our brothers? Analysis and detection of religious hate speech in the Arabic Twittersphere. In Proceedings of the 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. ACM. find
  3. Hend Al-Khalifa, Nadia Ghezaiel, Maria Bounnit, Hend Hamed Alhazmi, Noof Abdullah Alfear, Reem Fahad Alqifari, Ameera Masoud Almasoud, and Sharefah Ahmed Al-Ghamdi. 2026. ADAB: Arabic dataset for automated politeness benchmarking, a large-scale resource for computational sociopragmatics. In Proceedings of the Fifteenth biennial Language Resources and Evaluation Conference (LREC 2026), Palma, Mallorca, Spain. European Language Resources Association. To appear. ↗
  4. Hamdy Mubarak, Kareem Darwish, and Walid Magdy. 2017. Abusive language detection on Arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56. Association for Computational Linguistics.
  5. Ron Artstein and Massimo Poesio. 2008. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596.
  6. Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, and Halima Alshabani. 2019. L-HSAB: A Levantine Twitter dataset for hate speech and abusive language. In Proceedings of the Third Workshop on Abusive Language Online, pages 111–118. Association for Computational Linguistics.
  7. Kareem Darwish, Hamdy Mubarak, Ahmed Abdelali, and Mohamed Eldesouki. 2017. Arabic POS tagging: Don’t abandon feature engineering just yet. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 130–137. Association for Computational Linguistics.
  8. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media (ICWSM), pages 512–515. ↗
  9. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143, Online. Association for Computational Linguistics.
  10. Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of Twitter abusive behavior. In Proceedings of the 12th International AAAI Conference on Web and Social Media (ICWSM). ↗
  11. Ossama Obeid, Go Inoue, and Nizar Habash. 2022. CAMeLira: An Arabic multi-dialect morphological disambiguator. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 319–326, Abu Dhabi, UAE. Association for Computational Linguistics.
  12. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗
  13. Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, and Barbara Plank. 2023. What comes next? Evaluating uncertainty in neural text generators against human production variability. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14349–14371, Singapore. Association for Computational Linguistics. find
  14. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507–511. Association for Computational Linguistics.
  15. Nizar Habash. 2010. Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. ↗
  16. Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175–190, Seattle, United States. Association for Computational Linguistics. ↗
  17. Ahmed Haj Ahmed, Rui-Jie Yew, Xerxes Minocher, and Suresh Venkatasubramanian. 2025. Navigating dialectal bias and ethical complexities in Levantine Arabic hate speech detection. In Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4), pages 103–108, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  18. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275–9293. Association for Computational Linguistics. find
  19. Nan-Jiang Jiang, Chenhao Tan, and Marie-Catherine de Marneffe. 2023. Ecologically valid explanations for label variation in NLI. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10622–10633, Singapore. Association for Computational Linguistics.
  20. Salam Khalifa, Nizar Habash, Dana Abdulrahim, and Sara Hassan. 2016. A large scale corpus of Gulf Arabic. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 4282–4289.
  21. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385– 1470. ↗
  22. Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2023. Annotation error detection: Analyzing the past and present for a more coherent future. Computational Linguistics, 49(1):157–198. ↗
  23. Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019. Challenges and frontiers in abusive content detection. In Proceedings of the Third Workshop on Abusive Language Online, pages 80–99. Association for Computational Linguistics.
  24. Bertie Vidgen, Trista Thrush, Zeerak Waseem, and Douwe Kiela. 2021. Learning from the worst: Dynamically generated datasets to improve online hate detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1667–1682. Association for Computational Linguistics.
  25. Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter. In Proceedings of the NAACL Student Research Workshop, pages 88–93. Association for Computational Linguistics.
  26. Leon Weber-Genzel, Siyao Peng, Marie-Catherine de Marneffe, and Barbara Plank. 2024. VariErr NLI: Separating annotation error from human label variation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2256–2269. Association for Computational Linguistics.
  27. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. SemEval-2020 task 12: Multilingual offensive language identification in social media (Off-ComEval). In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1425–1447. International Committee for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaghouani-2026-beyond-annotator-disagreement-guideline-induced/