Elamur Research⌕
☰
home / papers / 2025 / Navigating Dialectal Bias and Ethical Complexities in Levantine Arabic Hate Speech Detection

Navigating Dialectal Bias and Ethical Complexities in Levantine Arabic Hate Speech Detection

Paper · 2025 · Ahmed Haj Ahmed, Rui-Jie Yew, Xerxes Minocher et al.

Summary

This position paper examines why hate speech detection works poorly for Levantine Arabic, a dialect continuum spoken in Syria, Jordan, Palestine and Lebanon. The authors describe regional differences in vocabulary and pronunciation. They also show how sociopolitical conflict can turn small linguistic features into markers of identity. For example, the way a speaker pronounces qaf became a sign of political allegiance in Syria. They then review the available resources. L-HSAB is the only public Levantine hate speech dataset, and the authors show that its keyword-based collection makes it heavily Lebanese and political. To test text representations, they train Logistic Regression and linear SVC classifiers on L-HSAB using several features: BoW, TF-IDF, Arabic fastText, a Word2Vec model trained on a small Levantine corpus, GloVe, and GoogleNews Word2Vec. Sparse features and Arabic-specific embeddings reach F1 scores of about 0.65–0.72. English-trained embeddings score close to zero. The paper argues for dialect-diverse, culturally informed datasets and models, because skewed data can wrongly flag harmless dialect speech and miss real hate speech in underrepresented varieties.

Key points

  • L-HSAB (Mulki et al., 2019) leans strongly toward Lebanese discourse. "Gebran Bassil" appears more than 1,000 times, "Lebanon" about 250 times, and "Wiam Wahhab" about 200 times.
  • Insults carry different weight in different regions. "za'ran" (thugs) is a strong insult in Lebanon but weaker in Syria. "shabbiha" (pro-regime militias) is loaded in Syria but may mean little to Lebanese readers.
  • The best F1 on L-HSAB came from TF-IDF with SVC (0.7217), followed by BoW unigrams with Logistic Regression (0.7177). Arabic fastText reached 0.6823 (LR) and 0.6964 (SVC).
  • Custom Word2Vec scored only 0.4429 and 0.3527. Its training corpus was about 21,959 words. GloVe scored about 0.06, and GoogleNews Word2Vec scored 0.0.
  • As next steps, the authors point to three morphologically annotated Levantine corpora, among them Baladi (Lebanese, ~9.6K tokens).

Interesting ideas

  • I1. Rank the most frequent named entities in a corpus to expose sampling bias left by keyword-driven collection; this audit is cheap and works on any corpus.
  • I2. Use embeddings trained on the wrong language as a near-zero baseline, which shows how much of a score depends on the representation actually matching the data.
  • I3. The same word can have very different offensive force depending on the speaker's community, so the severity of an insult is not a fixed property of the word.
  • I4. A phonetic or spelling variant that is neutral in one period can become an identity marker in another, so erasing variants during normalization can throw away meaning.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Run an entity and topic frequency audit on silver data and training treebanks to catch domain skew before it shapes the parser or the repair rules.
  • I2 (learn): Add mismatched-resource runs as explicit negative controls in evaluation, so that a component which "works" without real linguistic signal gets caught.
  • I3 (frames): Speech-act and irony judgments in prag could take speaker or community context as an input, rather than giving a lexical item one global connotation.
  • I4 (tokens): When the tokenizer and lemmatizer collapse dialectal or spelling variants, keep the original variant as a feature so later stages can still use it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Brown UniversityProvidence · United States80 works · 26,036 citations🎓Haverford CollegeHaverford · United States3 works · 457 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksSemitic languages · 3,066 worksBias, fairness & safety · 587 works

Cite

Cite as Ahmed Haj Ahmed, Rui-Jie Yew, Xerxes Minocher and Suresh Venkatasubramanian. 2025. Navigating Dialectal Bias and Ethical Complexities in Levantine Arabic Hate Speech Detection. Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ahmed2025navigating,
  title = {{Navigating Dialectal Bias and Ethical Complexities in Levantine Arabic Hate Speech Detection}},
  author = {Ahmed Haj Ahmed, Rui-Jie Yew, Xerxes Minocher and Suresh Venkatasubramanian},
  booktitle = {Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4)},
  year = {2025},
  url = {https://aclanthology.org/2025.wacl-1.12/}
}

References

29 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. ras (Palestinian, ~56K tokens), and Nabra (Syrian, ~60K tokens)—which we plan to combine into a more comprehensive Levantine corpus of approximately 125.6K tokens (Al-Haff et al., 2022; Nayouf et al., 2023). We expect that training our Word2Vec model on this expanded corpus will significantly enhance its performance. Additionally, while we discuss dialectal variations across Syria, Jordan, Palestine, and Lebanon, the linguistic analysis is not exhaustive, and some regional nuances may not be fully represented. Although we reference frameworks such as the playbook by Bergman and Diab (2022), we do not offer a detailed roadmap for creating inclusive and effective hate speech detection models. Future work should therefore focus on both enriching the training data resources and developing concrete tools to operationalize these recommendations, ensuring more accurate, contextually aware, and inclusive hate speech detection in Levantine Arabic. find
  2. This paper offers a conceptual discussion on the challenges of detecting hate speech in Levantine Arabic. While we provide experimental results to assess the effectiveness of different embedding techniques, the practical impact of our recommendations is still limited by the size and representativeness of our training data. Specifically, the custom dataset-trained Word2Vec model produced relatively lower F1 scores than the pre-trained Arabic fastText model, primarily due to the very limited size of our training corpus (approximately 21,959 words). We anticipate that performance would improve substantially with a larger Levantine Ara-References bic corpus. To this end, we have identified three promising morphologically annotated Levantine Monirah A. Al-Ajlan and Mourad Ykhlef. 2018. Opti-corpora—Baladi (Lebanese, ~9.6K tokens), Cur-mized twitter cyberbullying detection based on deep 107 learning. In 2018 21st Saudi Computer Society National Computer Conference (NCC), pages 1–5. find
  3. Karim Al-Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022. Curras + baladi: Towards a Levantine corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 769–778, Marseille, France. European Language Resources Association. ↗
  4. Azalden Alakrot, Liam Murray, and Nikola S. Nikolov. 2018. Dataset construction for the detection of antisocial behaviour in online communication in arabic. Procedia Computer Science, 142:174–181. Arabic Computational Linguistics. find
  5. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2020. Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In Proceedings of the 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM ’18, page 69–76. IEEE Press. find
  6. Imran Awan. 2014. Islamophobia and twitter: A typology of online hate against muslims on social media. Policy & Internet, 6(2):133–150. ↗
  7. Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2023. Fairness and Machine Learning: Limitations and Opportunities. MIT Press. find
  8. Emily Bender. 2019. The benderrule: On naming the languages we study and why it matters. The Gradient. find
  9. A. Stevie Bergman and Mona T. Diab. 2022. Towards responsible natural language annotation for the varieties of arabic. Preprint, arXiv:2203.09597. ↗
  10. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606. ↗
  11. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. ↗
  12. Sergio Andrés Castaño-Pulgarín, Natalia Suárez-Betancur, Luz Magnolia Tilano Vega, and Harvey Mauricio Herrera López. 2021. Internet, social media and online hate speech. systematic review. Aggression and Violent Behavior, 58:101608. ↗
  13. Zellig S. Harris. 1954. Distributional structure. find
  14. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. Preprint, arXiv:1301.3781. ↗
  15. Hamdy Mubarak, Kareem Darwish, and Walid Magdy. 2017. Abusive language detection on Arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56, Vancouver, BC, Canada. Association for Computational Linguistics.
  16. Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, and Halima Alshabani. 2019. L-HSAB: A Levantine Twitter dataset for hate speech and abusive language. In Proceedings of the Third Workshop on Abusive Language Online, pages 111–118, Florence, Italy. Association for Computational Linguistics.
  17. Amal Nayouf, Tymaa Hammouda, Mustafa Jarrar, Fadi Zaraket, and Mohamad-Bassam Kurdy. 2023. Nâbra: Syrian Arabic dialects with morphological annotations. In Proceedings of ArabicNLP 2023, pages 12–23, Singapore (Hybrid). Association for Computational Linguistics. ↗
  18. Samia Naïm. 2012. 53. Dialects of the Levant, pages 920–935. De Gruyter Mouton, Berlin, Boston. find
  19. M. Obégi. 1971. The Phonemic System of a Lebanese Arabic Dialect: (microfilm). Theses (Dept. of Modern Languages). National Library of Canada. find
  20. Solon Barocas and Andrew D. Selbst. 2016. Big data’s disparate impact. California Law Review, 104(3):671–732. find
  21. Karim El Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022. Curras + baladi: Towards a levantine corpus. Preprint, arXiv:2205.09692. ↗
  22. Md Saroar Jahan and Mourad Oussalah. 2023. A systematic review of hate speech automatic detection using natural language processing. Neurocomputing, 546:126232. ↗
  23. Rasha Omran. 2021. The letter qaf and conflicting grievances. https://www.syria.tv/%D8%AD%D8% B1%D9%81-%D8%A7%D9%84%D9%82%D8%A7%D9%81-% D9%88%D8%A7%D9%84%D9%85%D8%B8%D9%84%D9% 88%D9%85%D9%8A%D8%A7%D8%AA-%D8%A7%D9%84% D9%85%D8%AA%D8%B9%D8%A7%D8%B1%D8%B6%D8%A9. ↗
  24. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
  25. Evani Radiya-Dixit and Miranda Bogen. 2024. Beyond english-centric ai: Lessons on community participation from non-english nlp groups. find
  26. Ahmad Khalaf Sakarna. 2005. The linguistic status of the modern jordanian dialects. Arabica, 52(4):522– 543. ↗
  27. Karen Sparck Jones. 1988. A statistical interpretation of term specificity and its application in retrieval, page 132–142. Taylor Graham Publishing, GBR. find
  28. The Oversight Board. 2024. Referring to designated dangerous individuals as “shaheed”. find
  29. Uğur Ümit Üngör. 2020. Shabbiha: Paramilitary groups, mass violence and social polarization in homs. Violence: An International Journal, 1(1):59–79. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ahmed-2025-navigating-dialectal-bias-ethical-complexities/