Elamur Research⌕
☰
home / papers / 2022 / Learning From Arabic Corpora But Not Always From Arabic Speakers: A Case Study of the Arabic Wikipedia Editions

Learning From Arabic Corpora But Not Always From Arabic Speakers: A Case Study of the Arabic Wikipedia Editions

Paper · 2022 · Saied Alshahrani, Esma Wali and Jeanna Matthews

Summary

Wikipedia is a standard source of training text for languages other than English. The authors argue that text written in a language is not necessarily text written by its speakers. They compare three Arabic Wikipedia editions (Modern Standard Arabic, Egyptian Arabic and Moroccan Arabic), with English Wikipedia as a reference point. Using a July 2022 snapshot from the Wikimedia Statistics service, they look at total pages, newly created pages and the most active editors. The Egyptian Arabic edition stands out. One registered account, not labelled as a bot, created more than a million content pages in under three years. It did so through shallow, template-based translation from English, and much of the resulting text was never really converted into the dialect and remains MSA. Language models trained on this material may absorb the perspective and biases of the English source instead of the culture of Egyptian speakers. The paper therefore asks practitioners to check how a corpus was produced before learning from it. It proposes a finer classification of editor types and releases its analysis code.

Key points

  • Context: of the more than 7000 spoken languages, only about 300 have a Wikipedia edition, and these vary widely in size and in articles per speaker.
  • Compared editions: Arabic (MSA), Egyptian Arabic and Moroccan Arabic, plus English as a benchmark. The measures used were the cumulative page count, the number of newly created pages and the most active editors, all drawn from stats.wikimedia.org in July 2022.
  • Main finding: a single registered user automated the creation of over 1 million Egyptian Arabic pages in less than 3 years. Many of these pages are effectively MSA rather than Egyptian dialect.
  • The authors discuss a sophisticated alternative, the English→MSA→Egyptian pivot systems with domain and dialect adaptation by Jeblee et al. (2014). Even good translation would not fix one problem: the choice of topics would still reflect the source community rather than native speakers.
  • Recommendations and next steps:
    • add a "registered user (automation-assisted)" editor category;
    • develop a representativeness metric for spotting auto-generated pages;
    • build a neural classifier of corpus representativeness;
    • compile a list of more authentic Egyptian Arabic corpora.
  • Code and data are available at github.com/Clarkson-Accountability-Transparency/Analysis-of-Arabic-Wikipedias.

Interesting ideas

  • I1. Activity metadata can reveal inorganic text without reading it, for example page-creation volume per editor compared against a well-resourced edition.
  • I2. A binary human/bot label hides a middle class of humans driving automation, and an explicit intermediate category exposes it.
  • I3. Template-driven generation yields text that looks plentiful but repeats a few skeletons, which inflates corpus size without adding linguistic variety.
  • I4. Translation can be accurate and still unrepresentative, because the source community decided which topics exist and what to emphasise.
  • I5. Representativeness should be documented as a property of a corpus alongside its size and language.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): before raw English text enters silver-data pipelines, flag sources whose authorship statistics look mass-produced, so the witness parsers are not agreeing on templated text.
  • I2 (learn): tag every training or evaluation sentence with three-way provenance (human, LLM teacher, human with automated assistance) and report scores per group as a kind of negative control.
  • I3 (tree): when building or extending treebanks, deduplicate sentences by their structural skeleton so a parser is not overtrained on a handful of constructions.
  • I4 (back): when evaluating translation through the meta-world, check fidelity and also whether the content selection carries the source's framing, and record this as an explainable note.
  • I5 (world): attach provenance and representativeness notes to knowledge compiled from Wikipedia or Wikidata, so facts drawn from bot-filled regions can be traced and weighted with caution.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Clarkson UniversityPotsdam · United States5 works · 54 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksCommunication · 97 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Saied Alshahrani, Esma Wali and Jeanna Matthews. 2022. Learning From Arabic Corpora But Not Always From Arabic Speakers: A Case Study of the Arabic Wikipedia Editions. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.34

@inproceedings{alshahrani2022learning,
  title = {{Learning From Arabic Corpora But Not Always From Arabic Speakers: A Case Study of the Arabic Wikipedia Editions}},
  author = {Saied Alshahrani, Esma Wali and Jeanna Matthews},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.34},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.34}
}

References

59 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hitham Abo-Bakr, Khaled Shaalan, and Ibrahim Ziedan. 2008. A Hybrid Approach for Converting Written Egyptian Colloquial Dialect into Diacritized Arabic. In The 6th International Conference on Informatics and Systems (INFOS2008), Cairo, Egypt. Faculty of Computers and Information, Faculty of Computers and Information. find
  2. Kamla Al-Mannai, Hassan Sajjad, Alaa Khader, Fahad Al Obaidli, Preslav Nakov, and Stephan Vogel. 2014. Unsupervised Word Segmentation Improves Dialectal Arabic to English Machine Translation. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 207– 216, Doha, Qatar. Association for Computational Linguistics. find
  3. M. Arnold, R. K. E. Bellamy, M. Hind, S. Houde, S. Mehta, A. Mojsilović, R. Nair, K. Natesan Ramamurthy, A. Olteanu, D. Piorkowski, D. Reimer, J. Richards, J. Tsay, and K. R. Varshney. 2019. Fact-Sheets: Increasing trust in AI services through supplier’s declarations of conformity. IBM Journal of Research and Development, 63(4/5):6:1–6:13. find
  4. Marzieh Babaeianjelodar, Stephen Lorenz, Josh Gordon, Jeanna Matthews, and Evan Freitag. 2020. Quantifying Gender Bias in Different Corpora. In Companion Proceedings of the Web Conference 2020, WWW ’20, page 752–759, New York, NY, USA. Association for Computing Machinery. find
  5. Ibrahim Badr, Rabih Zbib, and James Glass. 2008. Segmentation for English-to-Arabic Statistical Machine Translation. In Proceedings of ACL-08: HLT, Short Papers, pages 153–156, Columbus, Ohio. Association for Computational Linguistics. find
  6. Maher Asaad Baker. 2022. How I Wrote a Million Wikipedia Articles, 2 edition. BookRix GmbH Co. KG., Munich, Germany. find
  7. Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics, 6:587–604. ↗
  8. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery. find
  9. Pablo Beytía, Pushkal Agarwal, Miriam Redi, and Vivek K Singh. 2022. Visual Gender Biases in Wikipedia: A Systematic Evaluation across the Ten Most Spoken Languages. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 43–54. ↗
  10. Runa Bhattacharjee and Pau Giner. 2022. You can now use Google Translate to translate articles on Wikipedia. Last accessed on 2022-09-11. find
  11. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. Advances in neural information processing systems, 29. ↗
  12. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA. Curran Associates Inc. ↗
  13. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. ↗
  14. Yan Chen, Christopher Mahoney, Isabella Grasso, Esma Wali, Abigail Matthews, Thomas Middleton, Mariama Njie, and Jeanna Matthews. 2021. Gender Bias and Under-Representation in Natural Language Processing Across Human Languages. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, page 24–34, New York, NY, USA. Association for Computing Machinery. find
  15. Won Ik Cho, Jiwon Kim, Jaeyeong Yang, and Nam Soo Kim. 2021. Towards Cross-Lingual Generalization of Translation Gender Bias. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 449–457, New York, NY, USA. Association for Computing Machinery. find
  16. Noam Cohen. 2008. Open-Source Troubles in Wiki World. The New York Times. Last accessed on 2022-09-11. find
  17. Frances Corry, Hamsini Sridharan, Alexandra Sasha Luccioni, Mike Ananny, Jason Schultz, and Kate Crawford. 2021. The Problem of Zombie Datasets: A Framework For Deprecating Datasets. ArXiv, abs/2111.04424. ↗
  18. Emily Denton, Alex Hanna, Razvan Amironesei, Andrew Smart, Hilary Nicole, and Morgan Klaus Scheuerman. 2020. Bringing the People Back In: Contesting Benchmark Machine Learning Datasets. CoRR, abs/2007.07399. ↗
  19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805. ↗
  20. Ahmed El-Kholy and Nizar Habash. 2010. Orthographic and morphological processing for English–Arabic statistical machine translation. Machine Translation, 26:25–45. ↗
  21. Ali Farghaly and Khaled Shaalan. 2009. Arabic Natural Language Processing: Challenges and Solutions. ACM Transactions on Asian Language Information Processing, 8(4). ↗
  22. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for Datasets. Communications of the ACM, 64(12):86– 92. ↗
  23. Nizar Habash, Ryan Roth, Owen Rambow, Ramy Eskander, and Nadi Tomeh. 2013. Morphological Analysis and Disambiguation for Dialectal Arabic. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 426–432, Atlanta, Georgia. Association for Computational Linguistics.
  24. Ari Hautasaari. 2013. "Could someone please translate this?": activity analysis of wikipedia article translation by non-experts. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW ’13, page 945–954, New York, NY, USA. Association for Computing Machinery. find
  25. Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2020. The Dataset Nutrition Label: A Framework to Drive Higher Data Quality Standards. Data Protection and Privacy, Volume 12: Data Protection and Democracy. find
  26. Cheng-Mao Hsu, Cheng te Li, Diego Sáez-Trumper, and Yi-Zhan Hsu. 2021. WikiContradiction: Detecting Self-Contradiction Articles on Wikipedia. 2021 IEEE International Conference on Big Data (Big Data), pages 427–436. find
  27. Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. 2021. Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 560–575, New York, NY, USA. Association for Computing Machinery. find
  28. Serena Jeblee, Weston Feely, Houda Bouamor, Alon Lavie, Nizar Habash, and Kemal Oflazer. 2014. Domain and Dialect Adaptation for Machine Translation into Egyptian Arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 196–206, Doha, Qatar. Association for Computational Linguistics.
  29. Isaac Johnson. 2020. Analyzing Wikidata Transclusion on English Wikipedia. CoRR, abs/2011.00997. ↗
  30. Isaac Johnson and Emily A. Lescak. 2022. Considerations for Multilingual Wikipedia Research. ArXiv, abs/2204.02483. ↗
  31. Maria Lopez-Medel. 2021. Gender bias in machine translation: an analysis of Google Translate in English and Spanish. Academia.edu. ↗
  32. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, page 220–229, New York, NY, USA. Association for Computing Machinery. find
  33. John Mittermeier, Ricardo Correia, Rich Grenyer, Tuuli Toivonen, and Uri Roll. 2021. Using wikipedia to measure public interest in biodiversity and conservation. Conservation Biology, 35. ↗
  34. Emad Mohamed, Behrang Mohit, and Kemal Oflazer. 2012. Transforming Standard Arabic to Colloquial Arabic. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 176–180, Jeju Island, Korea. Association for Computational Linguistics.
  35. Sergiu Nisioi, Ella Rabinovich, Liviu P. Dinu, and Shuly Wintner. 2016. A Corpus of Native, Non-native and Translated Texts. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4197–4201, Portorož, Slovenia. European Language Resources Association (ELRA). find
  36. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  37. Fabio Petroni, Tim Rocktäschel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019. Language Models as Knowledge Bases? CoRR, abs/1909.01066. ↗
  38. Gözde Gül Şahin. 2022. To Augment or Not to Augment? A Comparative Study on Text Augmentation Techniques for Low-Resource NLP. Computational Linguistics, 48(1):5–42. find
  39. Hassan Sajjad, Kareem Darwish, and Yonatan Belinkov. 2013a. Translating Dialectal Arabic to English. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–6, Sofia, Bulgaria. Association for Computational Linguistics. find
  40. Hassan Sajjad, Francisco Guzmán, Preslav Nakov, Ahmed Abdelali, Kenton Murray, Fahad Al Obaidli, and Stephan Vogel. 2013b. QCRI at IWSLT 2013: experiments in Arabic-English and English-Arabic spoken language translation. In Proceedings of the 10th International Workshop on Spoken Language Translation: Evaluation Campaign, Heidelberg, Germany. find
  41. Wael Salloum and Nizar Habash. 2013. Dialectal Arabic to English machine translation: Pivoting through Modern Standard Arabic. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 348–358, Atlanta, Georgia. Association for Computational Linguistics.
  42. Khaled Shaalan, Sanjeera Siddiqui, Manar Alkhatib, and Azza Monem. 2018. Challenges in Arabic Natural Language Processing. World Scientific. ↗
  43. Stefanie Ullmann and Danielle Saunders. 2021. Google Translate is sexist. What it needs is a little gender-sensitivity training. Last accessed on 2022-09-11. find
  44. Rodolfo V Valentim, Giovanni Comarela, Souneil Park, and Diego Sáez-Trumper. 2021. Tracking Knowledge Propagation Across Wikipedia Languages. In ICWSM, pages 1046–1052. ↗
  45. Esma Wali, Yan Chen, Christopher Mahoney, Thomas Middleton, Marzieh Babaeianjelodar, Mariama Njie, and Jeanna Neefe Matthews. 2020. Is Machine Learning Speaking my Language? A Critical Look at the NLP-Pipeline Across 8 Human Languages. arXiv preprint arXiv:2007.05872. ↗
  46. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. ↗
  47. Wikimedia Foundation. 2022. Content Translation -Mediawiki. Last accessed on 2022-09-11. find
  48. Wikimedia Foundation. 2022a. Mediawiki Action API. Last accessed on 2022-09-11. find
  49. Wikimedia Foundation. 2022b. Wikimedia Statistics. Last accessed on 2022-09-11. find
  50. Wikimedia Foundation. 2022c. Wikistats Metrics Definition. Last accessed on 2022-09-11. find
  51. Wikipedia. 2022a. Founder of Wikipedia. Last accessed on 2022-09-11. find
  52. Wikipedia. 2022b. User: Hitomiakane. Last accessed on 2022-09-11. find
  53. Wikipedia. 2022c. Wikipedia – The Free Encyclopedia. Last accessed on 2022-09-11. find
  54. Wikipedia. 2022d. Wikipedia Article Depth. Last accessed on 2022-09-11. find
  55. Wikipedia. 2022e. Wikipedia: Bot Policy. Last accessed on 2022-09-11. find
  56. KayYen Wong, Miriam Redi, and Diego Sáez-Trumper. 2021. Wiki-Reliability: A Large Scale Dataset for Content Reliability on Wikipedia. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. find
  57. Eddie Yang and Margaret E. Roberts. 2021. Censorship of Online Encyclopedias: Implications for NLP Models. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 537–548, New York, NY, USA. Association for Computing Machinery. find
  58. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F. Zaidan, and Chris Callison-Burch. 2012. Machine Translation of Arabic Dialects. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 49–59, Montréal, Canada. Association for Computational Linguistics.
  59. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In 2015 IEEE International Conference on Computer Vision, pages 19–27.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alshahrani-2022-learning-arabic-corpora-but-not/