Elamur Research⌕
☰
home / papers / 2022 / Towards Responsible Natural Language Annotation for the Varieties of Arabic

Towards Responsible Natural Language Annotation for the Varieties of Arabic

Paper · 2022 · A. Bergman and Mona Diab

Summary

This position paper argues that Arabic NLP datasets are often built with too little attention to how different the Arabic varieties are, and that this can harm vulnerable communities. The authors describe the main features that make annotation hard. Arabic is polyglossic: Modern Standard Arabic (MSA) coexists with regional vernaculars that may be mutually unintelligible. Words and phrases can mean different things across regions, there is no standard dialect spelling, and code switching is frequent. They support the argument with a case study of hate-speech labels on Facebook content from ten Arab countries. Labels from annotators based in Morocco (AiM) were compared with labels from native country experts (CE). The AiM reviewers judged Moroccan content most accurately and more leniently. For other countries they appeared either to misread the content or to be stricter with out-group speech. From these observations the authors derive a playbook for responsible dataset creation in multidialectal languages. It stresses local expertise, annotator support and careful coverage choices. The work matters because labels flawed in these ways feed content-moderation systems with wide reach.

Key points

  • The study sampled about 3000 items per country from a hate-speech evaluation set. Ten countries were covered: the North African states of Morocco, Algeria and Libya, the Levantine and Mesopotamian states of Lebanon, Syria and Iraq, plus Egypt, Sudan, Yemen and Saudi Arabia. Each item got a binary "delete"/"ignore" label.
  • Agreement among the country experts was fairly high, from 64% (Egyptian) to 92% (Moroccan). The overlap samples for this measure ranged from 91 items (Saudi) to 975 items (Iraqi).
  • In every country set, most content was in the local dialect rather than MSA. MSA was most common in the Saudi set (31%) and the Algerian set (27%). The experts judged 74% of the Saudi Arabic samples to be non-Saudi.
  • Signal Detection Theory was used to estimate each group's implied decision threshold. It showed that AiM was significantly more lenient on Moroccan content than on most other countries, the exception being Saudi. With CE labels taken as ground truth, AiM reached its best accuracy on Moroccan content (87%).
  • Following Habash (2010), the paper uses six broad dialect groups: Iraqi, Levantine, Maghrebi, Gulf, Yemeni and Egyptian. It notes that these groups hide country-level differences, such as Algerian versus Moroccan.

Interesting ideas

  • I1. Estimating each annotator group's implied decision threshold with Signal Detection Theory separates a group being strict or lenient from it being accurate, and it applies to any labeler, human or machine.
  • I2. Before anyone labels content, an expert first confirms which variety and country an item actually comes from, and mismatched items are filtered out.
  • I3. The paper treats "false friends" as a phrase-level problem, not only a word-level one. A blessing in one region can be a curse in another, so meaning depends on who is speaking.
  • I4. The authors prefer to narrow coverage when they cannot guarantee understanding and support, rather than covering everything superficially.
  • I5. Taking an expert cohort as ground truth and comparing other annotators against it per subgroup reveals hidden in-group/out-group bias in labels.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Witness parsers or LLM teacher tiers could be scored this way per genre, to show which one systematically over- or under-applies a label before votes are merged into silver data.
  • I2 (tree): Treebank sentences could pass a dialect/register check before UD dialect conversion or parser training, so that out-of-variety text does not silently become noise in the training data.
  • I3 (frames): Speech-act and irony detection could take a speaker-variety feature, so that an idiom's force (wish versus insult) is decided explicitly and the decision stays explainable.
  • I4 (world): Compiled knowledge (verb classes, ontology entries) could carry explicit scope tags and abstain outside those scopes, instead of guessing for regional senses it does not model.
  • I5 (learn): Evaluation could report accuracy against a gold reference broken down by text source or variety, used as a negative control, so that any per-subgroup gaps in the silver labels show up.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as A. Bergman and Mona Diab. 2022. Towards Responsible Natural Language Annotation for the Varieties of Arabic. Findings of the Association for Computational Linguistics: ACL 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-acl.31

@inproceedings{bergman2022responsible,
  title = {{Towards Responsible Natural Language Annotation for the Varieties of Arabic}},
  author = {A. Bergman and Mona Diab},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.31},
  url = {https://doi.org/10.18653/v1/2022.findings-acl.31}
}

References

38 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent Anti-Muslim Bias in Large Language Models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, page 298–306, New York, NY, USA. Association for Computing Machinery.
  2. Enam Al-Wer and Uri Horesh. 2018. The Routledge Handbook of Arabic Sociolinguistics. Routledge. ↗
  3. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? Proceedings of FAccT. find
  4. Amy Bruckman. 2020. ‘Have You Thought About...’: Talking About Ethical Implications of Research. Communications of the ACM, 63(9):38–40. ↗
  5. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77–91. find
  6. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. ↗
  7. Emily Denton, Alex Hanna, Razvan Amironesei, Andrew Smart, and Hilary Nicole. 2021. On the Genealogy of Machine Learning Datasets: A Critical History of ImageNet. Big Data & Society, 8(2):20539517211035955. ↗
  8. Terrance DeVries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. 2019. Does Object Recognition Work for Everyone? CoRR, abs/1906.02659. ↗
  9. Nicholas Diakopoulos. 2016. Accountability in algorithmic decision making. Communications of the ACM, 59(2). ↗
  10. Facebook. 2020. How We Review Content. find
  11. Amnesty International. 2020. Human Rights in Middle East and North Africa – Review in 2020. find
  12. Chloé Bakalar, Renata Barreto, Stevie Bergman, Miranda Bogen, Bobbie Chern, Sam Corbett-Davies, Melissa Hall, Isabel Kloumann, Michelle Lam, Joaquin Quiñonero Candela, Manish Raghavan, Joshua Simons, Jonathan Tannen, Edmund Tong, Kate Vredenburgh, and Jiejing Zhao. 2021. Fairness On The Ground: Applying Algorithmic Fairness Approaches to Production Systems. CoRR, abs/2103.06172. ↗
  13. Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2019. Fairness and Machine Learning. fairml-book.org. http://www.fairmlbook.org. ↗
  14. Solon Barocas and Andrew D. Selbst. 2016. Big Data’s Disparate Impact. California Law Review, 104(3):671–732. find
  15. Emily Bender. 2019. The #BenderRule: On naming the languages we study and why it matters. The Gradient. find
  16. Tarleton Gillespie. 2020. Content moderation, AI, and the question of scale. Big Data & Society, 7(2):2053951720943234. ↗
  17. Nizar Habash. 2010. Introduction to Arabic Natural Language Processing. In Synthesis Lectures on Human Language Technologies. Morgan and Claypool Publishers. ↗
  18. Alexa Hagerty and Igor Rubinov. 2019. Global AI Ethics: A Review of the Social Impacts and Ethical Implications of Artificial Intelligence. CoRR, abs/1907.07892. ↗
  19. Deborah Hellman. 2019. Measuring Algorithmic Fairness. Criminal Procedure eJournal. find
  20. Human Rights Watch. 2017. Arab Gulf States: Assault on Online Activists: Leading Bahraini Rights Activist Sentenced to Two Years in Prison. find
  21. Fatemah Husain and Ozlem Uzuner. 2021. A Survey of Offensive Language Detection for the Arabic Language. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 20(1). ↗
  22. Gregory Johnsen. 2021. The end of Yemen. Brookings. find
  23. Donald Martin Jr, Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William Isaac. 2020. Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics. CoRR, abs/2005.07572. ↗
  24. Gretchen McCulloch. 2019. Because Internet: Understanding the New Rules of Language. Riverhead Books. find
  25. Syria Relief. 2021. The Destruction You Can’t See: A report into the prevalence of Post Traumatic Stress Disorder (PTSD) symptoms amongst IDPs and refugees from the Syrian conflict. find
  26. Eve M. Troutt Powell. 2003. A Different Shade of Colonialism: Egypt, Great Britain, and the Mastery of the Sudan. University of California Press. find
  27. James Vincent. 2020. Facebook is now using AI to sort content for quicker moderation. The Verge. find
  28. Sarah J. Mielke. 2016. Language diversity in ACL 2004 - 2016. find
  29. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* 2019, Atlanta, GA, USA, January 29-31, 2019, pages 220–229. ACM. find
  30. Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. 2021. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. CoRR, abs/2103.14749. ↗
  31. Aviv Ovadya and Jess Whittlestone. 2019. Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning. arXiv preprint arXiv:1907.11274. ↗
  32. Partnership on AI. 2021. Managing the risks of ai research: Six recommendations for responsible publication. find
  33. Carina Prunkl, Carolyn Ashurst, Markus Anderljung, Helena Webb, Jan Leike, and Allan Dafoe. 2021. Institutionalizing ethics in AI through broader impact requirements. Nature Machine Intelligence. ↗
  34. Sanja Rancic, Sandro Radovanovic, and Boris Deliba-sic. 2021. Investigating Oversampling Techniques for Fair Machine Learning Models. In Decision Support Systems XI: Decision Support Systems, Analytics and Technologies in Response to Global Crisis Management - 7th International Conference on Decision Support System Technology, ICDSST 2021, Loughborough, UK, May 26-28, 2021, Proceedings, volume 414 of Lecture Notes in Business Information Processing, pages 110–123. Springer. find
  35. Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, and Vinodkumar Prabhakaran. 2020. Non-portability of Algorithmic Fairness to India. find
  36. Joey Shea and Ruba al-Hassani. 2021. Hate Speech, Social Media and Political Violence in Iraq: Virtual Civil Society and Upheaval. find
  37. Clare Sullivan. 2016. Digital citizenship and the right to digital identity under international law. Comput. Law Secur. Rev., 32(3):474–481. ↗
  38. Ryan Suto. 2016. Foreign Policy in Focus. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bergman-2022-responsible-natural-language-annotation-varieties/