Elamur Research⌕
☰
home / papers / 2022 / The Effect of Arabic Dialect Familiarity on Data Annotation

The Effect of Arabic Dialect Familiarity on Data Annotation

Paper · 2022 · Ibrahim Abu Farha and Walid Magdy

Summary

Arabic spans many regional dialects that differ in vocabulary, morphology and syntax. Crowd-sourced annotators often label text in dialects they barely know, and this can quietly damage subjective labels. The authors measure this effect using the Arabic part of SemEval-2022 Task 6 (iSarcasmEval), where the writers of the texts supplied both the sarcasm label and the dialect label. These author-given labels serve as a trustworthy reference. Third-party annotators from several dialect backgrounds then labelled the same texts for sarcasm and dialect, and their answers were compared with the author labels. The comparison was broken down by the annotator's native dialect, the dialects they know, and gender.

Annotators recognise their own dialect best and often mix up dialects that are unfamiliar to them. Their sarcasm judgments are more reliable on dialects they speak or know. Female annotators did somewhat better than male ones. The practical conclusion is that Arabic dataset builders should assign texts to native speakers of the relevant dialect, especially for subjective tasks.

Key points

  • The data is the Arabic test set of iSarcasmEval subtask A: 1400 sentences, of which 200 are sarcastic and 1200 are not. Each sentence carries author-provided labels for sarcasm and for one of five dialect classes: Nile Basin/Egyptian, Levantine, Gulf, Maghrebi, MSA.
  • Sarcasm performance is reported as F1 on the sarcastic class, sliced in three ways:
    • by the annotator's dialect,
    • by whether the annotator identified the text's dialect correctly,
    • by gender.
  • Egyptian and MSA were the easiest varieties to recognise in writing. Annotators tended to confuse dialects they were unfamiliar with.
  • The annotator pool was small; only one Gulf speaker took part, which the authors list as the main limitation.
  • All collected labels are released publicly (GitHub: iabufarha/arabic-dialect-familiarity).

Interesting ideas

  • I1. Using labels supplied by the text's author as a reference lets you measure third-party annotator error directly, rather than only measuring agreement between annotators.
  • I2. Annotators were asked to label the dialect alongside the main task, which shows whether a wrong task label comes from not recognising the variety at all.
  • I3. Task accuracy was broken down by the evaluator's familiarity with the variety, instead of being reported as one averaged score.
  • I4. Sarcasm was treated as dependent on local cultural references and dialect knowledge, not only on the literal surface.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data agreed by witness parsers could be checked against a small set where intent is known by construction, such as self-written or generated texts, so that shared bias among the witnesses becomes visible.
  • I2 (tree): The parser could predict the text's register or dialect (e.g. informal web English vs. edited prose) next to the UD tree. Errors could then be split by whether the register was guessed correctly, pointing to where dialect conversion is needed.
  • I3 (learn): Evaluation reports could stratify LAS and frame accuracy by text variety and by which witness or teacher produced the label. This would expose teachers that are only reliable on some registers.
  • I4 (frames): Irony detection in prag could record whether the needed background knowledge is available in the world model. When it is missing, the irony judgment should be marked as low-confidence rather than forced.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🔬The Alan Turing InstituteLondon · United Kingdom29 works · 4,368 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Ibrahim Abu Farha and Walid Magdy. 2022. The Effect of Arabic Dialect Familiarity on Data Annotation. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.39

@inproceedings{farha2022effect,
  title = {{The Effect of Arabic Dialect Familiarity on Data Annotation}},
  author = {Ibrahim Abu Farha and Walid Magdy},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.39},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.39}
}

References

41 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 296–305, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  2. Nora Al-Twairesh, Rawan Al-Matham, Nora Madi, Nada Almugren, Al-Hanouf Al-Aljmi, Shahad Alshalan, Raghad Alshalan, Nafla Alrumayyan, Shams Al-Manea, Sumayah Bawazeer, Nourah Al-Mutlaq, Nada Almanea, Waad Bin Huwaymil, Dalal Alqusair, Reem Alotaibi, Suha Al-Senaydi, and Abeer Alfutamani. 2018. Suar: Towards building a corpus for the saudi dialect. Procedia Computer Science, 142:72–82. Arabic Computational Linguistics. find
  3. Hassan Alhuzali, Muhammad Abdul-Mageed, and Lyle Ungar. 2018. Enabling deep learning of emotion with first-person seed expressions. In Proceedings of the Second Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media, pages 25–35, New Orleans, Louisiana, USA. Association for Computational Linguistics.
  4. Nizar Habash, Owen Rambow, Mona Diab, and Reem Kanjawi-Faraj. 2008. Guidelines for annotation of arabic dialectness. In Proceedings of the LREC Workshop on HLT & NLP within the Arabic world, pages 49–53. find
  5. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mario J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In CoNLL, pages 167–177. ACL. ↗
  6. David Bamman and Noah A. Smith. 2015. Contextualized sarcasm detection on twitter. In ICWSM, pages 574–577. AAAI Press. find
  7. Nizar Y Habash. 2010. Introduction to Arabic Natural Language Processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. ↗
  8. Salima Harrat, Karima Meftouh, Mourad Abbas, and Kamel Smaïli. 2014. Building Resources for Algerian Arabic Dialects. In 15th Annual Conference of the International Communication Association Interspeech, Singapour, Singapore. ISCA. find
  9. Francesco Barbieri, Francesco Ronzano, and Horacio Saggion. 2014. Italian irony detection in twitter: a first approach. In CLiC-it, page 28. AILC. ↗
  10. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. Cascade: Contextual sarcasm detection in online discussion forums. In COLING, pages 1837– 1848. ACL.
  11. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1240–1245, Reykjavik, Iceland. European Language Resources Association (ELRA).
  12. Mustafa Jarrar, Nizar Habash, Faeq Alrimawi, Diyam Akra, and Nasser Zalmout. 2017. Curras: an annotated corpus for the palestinian arabic dialect. Language Resources and Evaluation, 51(3):745–775. ↗
  13. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207, Florence, Italy. Association for Computational Linguistics.
  14. John D Campbell and Albert N Katz. 2012. Are there necessary conditions for inducing a sense of sarcastic irony? Discourse Processes, 49(6):459–480. ↗
  15. Ryan Cotterell and Chris Callison-Burch. 2014. A multi-dialect, multi-genre corpus of informal written Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 241–245, Reykjavik, Iceland. European Language Resources Association (ELRA).
  16. Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep models for Arabic dialect identification on benchmarked data. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 263–274, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  17. Heba Elfardy, Mohamed Al-Badrashiny, and Mona Diab. 2014. AIDA: Identifying code switching in informal Arabic text. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 94–101, Doha, Qatar. Association for Computational Linguistics.
  18. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In LREC. ELRA. ↗
  19. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016. Are word embedding-based features useful for sarcasm detection? In EMNLP, pages 1006–1011. ACL. ↗
  20. Salam Khalifa, Nizar Habash, Dana Abdulrahim, and Sara Hassan. 2016. A large scale corpus of Gulf Arabic. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4282–4289, Portorož, Slovenia. European Language Resources Association (ELRA).
  21. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2018. A large self-annotated corpus for sarcasm. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  22. Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Ophir Frieder. 2019. Hate speech detection: Challenges and solutions. PLOS ONE, 14(8):1–16. ↗
  23. Walaa Medhat, Ahmed Hassan, and Hoda Korashy. 2014. Sentiment analysis algorithms and applications: A survey. Ain Shams Engineering Journal, 5(4):1093–1113. ↗
  24. Hamdy Mubarak and Kareem Darwish. 2014. Using Twitter to collect a multi-dialectal corpus of Arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 1–7, Doha, Qatar. Association for Computational Linguistics.
  25. Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020. Overview of OSACT4 Arabic offensive language detection shared task. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 48–52, Marseille, France. European Language Resource Association.
  26. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332– 1344, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  27. Silviu Oprea and Walid Magdy. 2019. Exploring author context for detecting intended vs perceived sarcasm. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2854– 2859, Florence, Italy. Association for Computational Linguistics. ↗
  28. Byron C. Wallace, Do Kook Choe, and Eugene Charniak. 2015. Sparse, contextually informed models for irony detection: Exploiting user communities, entities and sentiment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1035–1044, Beijing, China. Association for Computational Linguistics. ↗
  29. Silviu Oprea and Walid Magdy. 2020a. iSarcasm: A dataset of intended sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1279–1289, Online. Association for Computational Linguistics.
  30. Silviu Vlad Oprea and Walid Magdy. 2020b. The effect of sociocultural variables on sarcasm communication online. Proc. ACM Hum.-Comput. Interact., 4(CSCW1). ↗
  31. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507–511, Baltimore, Maryland. Association for Computational Linguistics. ↗
  32. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In COL-ING, pages 213–223. ACL.
  33. Veranika Puhacheuskaya and Juhani Järvikivi. 2022. I was being sarcastic!: The effect of foreign accent and political ideology on irony (mis) understanding. Acta Psychologica, 222:103479. ↗
  34. Byron C. Wallace, Do Kook Choe, Laura Kertz, and Eugene Charniak. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 512–516, Baltimore, Maryland. Association for Computational Linguistics. ↗
  35. Deirdre Wilson. 2006. The pragmatics of verbal irony: Echo or pretence? Lingua, 116(10):1722–1743. ↗
  36. Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41, Portland, Oregon, USA. Association for Computational Linguistics.
  37. Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic Dialect Identification. Computational Linguistics, 40(1):171–202. ↗
  38. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In WSDM, pages 97–106. ACM. ↗
  39. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In EMNLP, pages 704–714. ACL. ↗
  40. Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175–190, Seattle, United States. Association for Computational Linguistics. ↗
  41. Fatiha Sadat, Farzindar Kazemi, and Atefeh Farzindar. 2014. Automatic identification of Arabic language varieties and dialects in social media. In Proceedings of the Second Workshop on Natural Language Processing for Social Media (SocialNLP), pages 22– 27, Dublin, Ireland. Association for Computational Linguistics and Dublin City University.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/farha-2022-effect-arabic-dialect-familiarity-data/