Elamur Research⌕
☰
home / papers / 2024 / Estimating the Level of Dialectness Predicts Interannotator Agreement in Multi-dialect Arabic Datasets

Estimating the Level of Dialectness Predicts Interannotator Agreement in Multi-dialect Arabic Datasets

Summary

Multi-dialect Arabic datasets are usually labeled by native Arabic speakers who receive samples at random, with no regard to which dialect each speaker knows. Earlier studies suggested sending dialectal samples to speakers of the matching dialect. That is hard in practice: automatic dialect identification is unreliable, and annotators for some dialects are scarce. The authors ask whether an automatic dialectness score can show which samples actually need such routing.

They score every sample with the Sentence-ALDi model, which rates divergence from Modern Standard Arabic on a 0–1 scale. They then relate this score to how often all annotators chose the same label. The study covers 15 dataset–task combinations from public corpora that release individual annotations.

For most non-dialect-ID tasks, agreement falls as dialectness rises. For dialect identification, agreement rises instead. The practical upshot is a cheaper workflow. Near-MSA text can go to any Arabic speaker. Only highly dialectal text needs dialect-matched annotators, and for those samples automatic dialect ID should also work better.

Key points

  • Dataset search: Candidates came from the Masader catalog. The authors inspected 151 non-MSA-only entries and found 28 suitable candidates. After asking authors for raw labels, they kept 15 dataset–task pairs from 9 corpora: DCD, MPOLD, YTCB, ASAD, ArSAS, ArSarcasm-v1, Mawqif, iSarcasm, and DART.
  • Tasks covered: offensive/hate speech, sarcasm, sentiment, speech act, stance, and dialect identification.
  • Method:
    • Samples were binned into 10 equal ALDi bins of width 0.1.
    • The metric was "% full agree" per bin, meaning the share of samples where every annotator gave the same label.
    • Pearson correlation measured the relationship, and the slope of a fitted line gave the effect size.
    • Results stayed qualitatively the same with 4 or 20 bins and with per-sample logistic regression.
  • Main result: 11 of the 15 cases support the hypothesis.
    • 8 of the 12 non-DI datasets show a significant negative correlation, with slopes as steep as about −41 points across the ALDi range.
    • The 3 DI datasets show higher agreement for more dialectal text.
  • Share of near-MSA text: In 6 datasets, more than half the samples score below 0.1 ALDi, meaning they are essentially MSA. The trends hold even when these samples are removed.

Interesting ideas

  • I1. Treat "how far a sentence is from the standard variety" as a continuous, automatically estimated difficulty variable rather than a categorical dialect label. It is cheaper and more robust than forcing one dialect tag on each text. — For Mova (tree): A non-standardness score per sentence (slang, informal or dialectal English) could decide which inputs get the UD dialect conversion path or the heavier gparse parser, instead of treating all text the same.
  • I2. Use the score to route work: easy, near-standard items go to any worker, and only high-divergence items go to specialized annotators. This cuts cost without losing quality. — For Mova (learn): Silver data could send low-divergence sentences to the cheap witness-parser vote and reserve LLM teacher tiers for high-divergence sentences, where witnesses are likely to disagree.
  • I3. Plot agreement against a difficulty score in bins, report correlation plus slope, and check robustness with other bin counts and a logistic regression. This is a simple, transparent audit of where labels become unreliable. — For Mova (learn): The same binned audit could be run on witness-parser agreement against sentence features (length, informality, rare words) to find where silver trees should not be trusted.
  • I4. Split agreement trends by class. Negative labels ("not sarcastic", "neutral") need full comprehension, while positive labels can be triggered by a single cue word. The paper also notes that agreement driven by cues is not the same as being correct. — For Mova (frames): Irony and speech-act decisions should record whether they came from full understanding of the frame or from a lone lexical cue, so explanations can flag cue-only judgments as weaker.
  • I5. Remove the trivial region (here, near-MSA samples) and re-check that the trend survives. This guards against a result driven by a dominant easy subset. — For Mova (learn): Evaluations could routinely drop the easiest bucket of test items as a control, to confirm that reported gains are not carried by trivially simple sentences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.11282 · arXiv 2405.11282

@misc{keleg2024estimating,
  title = {{Estimating the Level of Dialectness Predicts Interannotator Agreement in Multi-dialect Arabic Datasets}},
  author = {Keleg, Amr and Magdy, Walid and Goldwater, Sharon},
  year = {2024},
  doi = {10.48550/arxiv.2405.11282},
  eprint = {2405.11282},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.11282}
}

References

30 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, El Moatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023. NADI 2023: The fourth nuanced Arabic dialect identification shared task. In Proceedings of ArabicNLP 2023, pages 600– 613, Singapore (Hybrid). Association for Computational Linguistics. find
  2. Ibrahim Abu Farha and Walid Magdy. 2020. From Arabic sentiment analysis to sarcasm detection: The ArSarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39, Marseille, France. European Language Resource Association. find our card
  3. Ibrahim Abu Farha and Walid Magdy. 2022. The effect of Arabic dialect familiarity on data annotation. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 399– 408, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  4. Ibrahim Abu Farha, Silviu Vlad Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 task 6: iSarcasmEval, intended sarcasm detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 802–814, Seattle, United States. Association for Computational Linguistics. find our card
  5. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 296–305, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find our card
  6. Azalden Alakrot, Liam Murray, and Nikola S. Nikolov. 2018. Dataset construction for the detection of antisocial behaviour in online communication in Arabic. Procedia Computer Science, 142:174–181. Arabic Computational Linguistics. find
  7. Basma Alharbi, Hind Alamro, Manal Alshehri, Zuhair Khayyat, Manal Kalkatawi, Inji Ibrahim Jaber, and Xiangliang Zhang. 2021. Asad: A twitter-based benchmark Arabic sentiment analysis dataset. find our card
  8. Israa Alsarsour, Esraa Mohamed, Reem Suwaileh, and Tamer Elsayed. 2018. DART: A large dataset of dialectal Arabic tweets. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  9. Yousef Altaher, Ali Fadel, Mazen Alotaibi, Zaid Alyazidi, et al. 2022. Masader Plus: A new interface for exploring +500 Arabic NLP datasets. arXiv preprint arXiv:2208.00932. ↗ our card
  10. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label Arabic dataset for target-specific stance detection. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  11. Zaid Alyafeai, Maraim Masoud, Mustafa Ghaleb, and Maged S. Al-shaibani. 2021. Masader: Metadata sourcing for Arabic text and speech data resources. find our card
  12. As-Said Muhámmad Badawi. 1973. Mustawayat al-arabiyya al-muasira fi Misr. Dar al-maarif. find
  13. A. Bergman and Mona Diab. 2022. Towards responsible natural language annotation for the varieties of Arabic. In Findings of the Association for Computational Linguistics: ACL 2022, pages 364–371, Dublin, Ireland. Association for Computational Linguistics. find
  14. Business for Social Responsibility. 2022. Human rights due diligence of meta’s impacts in Israel and Palestine in may 2021. https://about.fb.com/wp-content/uploads/2022/09/Human-Rights-Due-Diligence-of-Metas-Impacts-in-Israel-and-Palestine-in-May-2021.pdf. ↗
  15. Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Abdelali, Soon-gyo Jung, Bernard J. Jansen, and Joni Salminen. 2020. A multi-platform Arabic news comment dataset for offensive language detection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6203–6212, Marseille, France. European Language Resources Association. find our card
  16. AbdelRahim A. Elmadany, Hamdy Mubarak, and Walid Magdy. 2018. An Arabic speech-act and sentiment corpus of tweets. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA). The 3rd Workshop on Open-Source Arabic Corpora and Processing Tools, OSACT3 ; Conference date: 08-05-2018. find
  17. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. ↗
  18. Nizar Y. Habash. 2010. Introduction to Arabic natural language processing, 1 edition, volume 3 of Synthesis Lectures on Human Language Technologies. Morgan and Claypool Publishers. find
  19. Amr Keleg, Sharon Goldwater, and Walid Magdy. 2023. ALDi: Quantifying the Arabic level of dialectness of text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10597–10611, Singapore. Association for Computational Linguistics. find
  20. Amr Keleg and Walid Magdy. 2023. Arabic dialect identification under scrutiny: Limitations of single-label classification. In Proceedings of ArabicNLP 2023, pages 385–398, Singapore (Hybrid). Association for Computational Linguistics. find
  21. Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, Nicolas Rey-Villamizar, Mona Diab, and Thamar Solorio. 2016. Overview for the second shared task on language identification in code-switched data. In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 40–49, Austin, Texas. Association for Computational Linguistics. find
  22. Hamdy Mubarak and Kareem Darwish. 2016. Demographic surveys of Arab annotators on CrowdFlower. In Proceedings of ACM WebSci16 Workshop “Weaving Relations of Trust in Crowd Work: Transparency and Reputation across Platforms. find
  23. Hamdy Mubarak, Kareem Darwish, and Walid Magdy. 2017. Abusive language detection on Arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56, Vancouver, BC, Canada. Association for Computational Linguistics. find our card
  24. Mahmoud Nabil, Mohamed Aly, and Amir Atiya. 2015. ASTD: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515–2519, Lisbon, Portugal. Association for Computational Linguistics. find our card
  25. Helene Olsen, Samia Touileb, and Erik Velldal. 2023. Arabic dialect identification: An in-depth error analysis on the MADAR parallel corpus. In Proceedings of ArabicNLP 2023, pages 370–384, Singapore (Hybrid). Association for Computational Linguistics. find
  26. Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit-Yan Yeung. 2019. Multilingual and multi-aspect hate speech analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4675– 4684, Hong Kong, China. Association for Computational Linguistics. find our card
  27. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502– 518, Vancouver, Canada. Association for Computational Linguistics. find our card
  28. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884–5906, Seattle, United States. Association for Computational Linguistics. find our card
  29. Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. find
  30. Howard Wainer, Marc Gessaroli, and Monica Verdi. 2006. Visual revelations. CHANCE, 19(1):49–52. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/keleg-2024-estimating-level-dialectness-predicts-interannotator/