Elamur Research⌕
☰
home / papers / 2024 / Estimating the Level of Dialectness Predicts Inter-annotator Agreement in Multi-dialect Arabic Datasets

Estimating the Level of Dialectness Predicts Inter-annotator Agreement in Multi-dialect Arabic Datasets

Paper · 2024 · Amr Keleg, Walid Magdy and Sharon Goldwater

Summary

Multi-dialect Arabic datasets are usually labelled by native Arabic speakers assigned to samples at random. Earlier studies recommended sending each sample to speakers of its own dialect. That is hard in practice: automatic dialect identification is unreliable, and annotators for some dialects are scarce. The authors test whether the Arabic Level of Dialectness (ALDi) can guide where routing actually matters. ALDi is a 0–1 score of how far a sentence departs from MSA.

They scored samples from 15 public datasets with the Sentence-ALDi model. These datasets cover offensive language, hate speech, sarcasm, sentiment, speech acts, stance and dialect identification, and all of them release per-annotator labels. They then grouped samples into ALDi bins and measured how often all annotators agreed in each bin.

For most non-DI datasets, agreement falls as dialectness rises. For dialect identification, agreement rises with dialectness. The practical recommendation follows from this. Low-ALDi, MSA-like samples can go to any annotator. Dialect-specific routing should focus on high-ALDi samples, which are also the ones whose dialect is easier to detect automatically.

Key points

  • Data selection: datasets were found via the Masader catalog. The authors manually reviewed 151 non-MSA-only entries, shortlisted 28, and ended with 15 datasets that had individual labels, some obtained from the original authors.
  • Method:
    • Samples were binned into 10 bins of width 0.1 by Sentence-ALDi score.
    • The measure was "% full agree", the share of samples in a bin where all annotators chose the same label.
    • The relationship was quantified with Pearson correlation and the slope of a fitted line.
    • Robustness checks used 4 or 20 bins and a per-sample logistic regression.
  • Non-DI tasks: strong negative correlations appeared in 8 of 12 non-DI datasets (e.g., ASAD ρ≈−0.99).
  • DI tasks: all 3 DI datasets showed a positive correlation (e.g., DART ρ≈0.97).
  • MSA-heavy data: in 6 datasets, more than half of the samples score ALDi < 0.1, so these are essentially MSA.
  • By label: the drop in agreement is clearest for "negative" labels (not sarcastic, neutral, non-offensive). Positive labels can be agreed on from isolated cue words, such as an MSA insult inside a dialectal sentence.

Interesting ideas

  • I1. Use a cheap continuous "distance from the standard variety" score to decide which items need specialised annotators, instead of first solving full variety identification.
  • I2. Bin a continuous property, plot full-agreement rate per bin, report both slope and correlation, and confirm the pattern with a per-item logistic regression and different bin counts.
  • I3. Agreement on positive labels can come from a single salient cue word rather than real understanding, so high agreement does not prove a label is correct.
  • I4. Rejecting a label ("not sarcastic", "neutral") needs the whole sentence to be understood, so this is where comprehension gaps show up first.
  • I5. The same dialectness score works in opposite directions depending on the task: strongly dialectal text is harder to classify but easier to assign to a dialect.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): score sentences for informality or non-standardness and send only the high-scoring ones to the stronger LLM teacher tier, leaving standard text to witness-parser agreement.
  • I2 (learn): plot witness-parser full agreement against features such as sentence length or rarity of constructions, to see where silver data becomes unreliable.
  • I3 (frames): irony or offence detection should not fire on one lexical cue unless the rest of the sentence was actually parsed and understood.
  • I4 (answer): when a reader answers "no / not the case", check that no part of the text was left unparsed, and lower confidence when coverage is partial.
  • I5 (tree): in UD dialect conversion, run variety-specific rules only on strongly marked sentences, where detecting the variety is reliable, and treat near-standard ones with the default parser.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Amr Keleg, Walid Magdy and Sharon Goldwater. 2024. Estimating the Level of Dialectness Predicts Inter-annotator Agreement in Multi-dialect Arabic Datasets. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-short.70

@inproceedings{keleg2024estimating,
  title = {{Estimating the Level of Dialectness Predicts Inter-annotator Agreement in Multi-dialect Arabic Datasets}},
  author = {Amr Keleg, Walid Magdy and Sharon Goldwater},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-short.70},
  url = {https://doi.org/10.18653/v1/2024.acl-short.70}
}

References

30 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, El Moatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023. NADI 2023: The fourth nuanced Arabic dialect identification shared task. In Proceedings of ArabicNLP 2023, pages 600– 613, Singapore (Hybrid). Association for Computational Linguistics.
  2. Ibrahim Abu Farha and Walid Magdy. 2020. From Arabic sentiment analysis to sarcasm detection: The ArSarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39, Marseille, France. European Language Resource Association.
  3. Ibrahim Abu Farha and Walid Magdy. 2022. The effect of Arabic dialect familiarity on data annotation. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 399– 408, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  4. Ibrahim Abu Farha, Silviu Vlad Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 task 6: iSarcasmEval, intended sarcasm detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 802–814, Seattle, United States. Association for Computational Linguistics.
  5. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 296–305, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  6. Azalden Alakrot, Liam Murray, and Nikola S. Nikolov. 2018. Dataset construction for the detection of antisocial behaviour in online communication in Arabic. Procedia Computer Science, 142:174–181. Arabic Computational Linguistics. find
  7. Basma Alharbi, Hind Alamro, Manal Alshehri, Zuhair Khayyat, Manal Kalkatawi, Inji Ibrahim Jaber, and Xiangliang Zhang. 2021. Asad: A twitter-based benchmark Arabic sentiment analysis dataset. ↗
  8. A. Bergman and Mona Diab. 2022. Towards responsible natural language annotation for the varieties of Arabic. In Findings of the Association for Computational Linguistics: ACL 2022, pages 364–371, Dublin, Ireland. Association for Computational Linguistics.
  9. Business for Social Responsibility. 2022. Human rights due diligence of meta’s impacts in Israel and Palestine in may 2021. https://about.fb.com/wp-content/uploads/2022/09/Human-Rights-Due-Diligence-of-Metas-Impacts-in-Israel-and-Palestine-in-May-2021.pdf. ↗
  10. Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Abdelali, Soon-gyo Jung, Bernard J. Jansen, and Joni Salminen. 2020. A multi-platform Arabic news comment dataset for offensive language detection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6203–6212, Marseille, France. European Language Resources Association.
  11. AbdelRahim A. Elmadany, Hamdy Mubarak, and Walid Magdy. 2018. An Arabic speech-act and sentiment corpus of tweets. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA). The 3rd Workshop on Open-Source Arabic Corpora and Processing Tools, OSACT3 ; Conference date: 08-05-2018. find
  12. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. ↗
  13. Nizar Y. Habash. 2010. Introduction to Arabic natural language processing, 1 edition, volume 3 of Synthesis Lectures on Human Language Technologies. Morgan and Claypool Publishers. find
  14. Israa Alsarsour, Esraa Mohamed, Reem Suwaileh, and Tamer Elsayed. 2018. DART: A large dataset of dialectal Arabic tweets. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  15. Amr Keleg, Sharon Goldwater, and Walid Magdy. 2023. ALDi: Quantifying the Arabic level of dialectness of text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10597–10611, Singapore. Association for Computational Linguistics.
  16. Yousef Altaher, Ali Fadel, Mazen Alotaibi, Zaid Alyazidi, et al. 2022. Masader Plus: A new interface for exploring +500 Arabic NLP datasets. arXiv preprint arXiv:2208.00932. ↗
  17. Amr Keleg and Walid Magdy. 2023. Arabic dialect identification under scrutiny: Limitations of single-label classification. In Proceedings of ArabicNLP 2023, pages 385–398, Singapore (Hybrid). Association for Computational Linguistics.
  18. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label Arabic dataset for target-specific stance detection. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  19. Zaid Alyafeai, Maraim Masoud, Mustafa Ghaleb, and Maged S. Al-shaibani. 2021. Masader: Metadata sourcing for Arabic text and speech data resources.
  20. As-Said Muhámmad Badawi. 1973. Mustawayat al-arabiyya al-muasira fi Misr. Dar al-maarif. find
  21. Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, Nicolas Rey-Villamizar, Mona Diab, and Thamar Solorio. 2016. Overview for the second shared task on language identification in code-switched data. In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 40–49, Austin, Texas. Association for Computational Linguistics.
  22. Hamdy Mubarak and Kareem Darwish. 2016. Demographic surveys of Arab annotators on CrowdFlower. In Proceedings of ACM WebSci16 Workshop “Weaving Relations of Trust in Crowd Work: Transparency and Reputation across Platforms. find
  23. Hamdy Mubarak, Kareem Darwish, and Walid Magdy. 2017. Abusive language detection on Arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56, Vancouver, BC, Canada. Association for Computational Linguistics.
  24. Mahmoud Nabil, Mohamed Aly, and Amir Atiya. 2015. ASTD: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515–2519, Lisbon, Portugal. Association for Computational Linguistics. ↗
  25. Helene Olsen, Samia Touileb, and Erik Velldal. 2023. Arabic dialect identification: An in-depth error analysis on the MADAR parallel corpus. In Proceedings of ArabicNLP 2023, pages 370–384, Singapore (Hybrid). Association for Computational Linguistics.
  26. Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit-Yan Yeung. 2019. Multilingual and multi-aspect hate speech analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4675– 4684, Hong Kong, China. Association for Computational Linguistics.
  27. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502– 518, Vancouver, Canada. Association for Computational Linguistics.
  28. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884–5906, Seattle, United States. Association for Computational Linguistics. ↗
  29. Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. ↗
  30. Howard Wainer, Marc Gessaroli, and Monica Verdi. 2006. Visual revelations. CHANCE, 19(1):49–52. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/keleg-2024-estimating-level-dialectness-predicts-inter/