Elamur Research⌕
☰
home / papers / 2024 / Leveraging Corpus Metadata to Detect Template-based Translation: An Exploratory Case Study of the Egyptian Arabic Wikipedia Edition

Leveraging Corpus Metadata to Detect Template-based Translation: An Exploratory Case Study of the Egyptian Arabic Wikipedia Edition

Summary

The paper tackles a corpus-quality problem in the Egyptian Arabic Wikipedia (ARZ). A small number of registered users mass-produced over a million articles by running English pages through Google Translate with simple templates, and no human review followed. The resulting text is low in quality and does not reflect Egyptian culture or dialect, so it is a poor basis for training language models.

The authors first compare ARZ with the Arabic (AR) and Moroccan (ARY) editions, using January 2024 dumps and per-article metadata from the Wikimedia XTools API. They look at content density, lexical diversity, repeated n-grams and the share of edits made by humans versus bots. Heuristic rules then help build a labeled set of human-generated and template-translated articles. On that set they train multivariate classifiers on metadata and text embeddings.

The best model is XGBoost using metadata together with CAMeLBERT embeddings. It is deployed as a public web tool, the Egyptian Wikipedia Scanner, and the labeled datasets are released. The work gives researchers a practical way to remove inorganic text from dialectal corpora.

Key points

  • ARZ has more articles than AR (1,621,745 vs 1,226,784) but fewer usable ones after Gensim extraction (736,158 vs 880,334). Gensim drops texts under 50 tokens, which removed about 741K ARZ articles (46%), compared with 28% for AR and 30% for ARY.
  • ARZ articles are the shortest of the three editions, averaging about 610 characters and 100 tokens.
  • On lexical diversity:
    • Plain TTR gave identical values for AR and ARZ, which the authors treat as misleading.
    • MTLD, computed with the LexicalRichness library at factor 0.720, ranks ARY first, AR second and ARZ last.
  • Repeated n-grams with n up to 10 and 50 were mined to expose the translation templates. Heuristic filtering rules were also applied, and the paper reports how many articles each rule removed.
  • XGBoost with metadata plus CAMeLBERT embeddings performed best on accuracy and ROC-AUC:
    • Two ablation studies showed that metadata features help the classifier.
    • CAMeLBERT was preferred over averaged Word2Vec vectors because its embeddings are contextual.
    • The resulting app is hosted on Streamlit Cloud and Hugging Face Spaces and is MIT-licensed on GitHub.

Interesting ideas

  • I1. Edit history (edit counts, editor counts, creator, byte size) can reveal how an article was produced without reading the text at all, making it a cheap first-pass corpus filter.
  • I2. Counting very long duplicate n-grams (10- and 50-grams) is a simple way to surface boilerplate templates that short n-grams would miss.
  • I3. TTR-style ratios can give the same value for corpora that clearly differ, so a length-robust measure like MTLD is the safer choice.
  • I4. Labels were produced by a stack of heuristic rules, with the number of items caught by each rule reported. This makes the labeling transparent and auditable.
  • I5. Ablations that compare content-only features with content-plus-metadata features show whether the side signals truly add information.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): score raw English sources by provenance metadata before they are parsed into silver data, so machine-made or templated text never enters the training pool.
  • I2 (tree): run long-n-gram duplicate detection over treebank and silver sentences, then down-weight or deduplicate templated sentences so the parser does not overfit to them.
  • I3 (back): use MTLD rather than TTR to check whether text regenerated through the text → graph → text compressor keeps the lexical variety of the original.
  • I4 (learn): log per-rule hit counts for the self-induced repair rules, so each rule's contribution can be inspected and suspicious rules caught early.
  • I5 (learn): add similar ablations to evaluation, training with and without a given feature group alongside the negative controls, to confirm each feature group earns its place.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics Semitic languages

Cite

Cite as Saied Alshahrani, Hesham Haroon Mohammed, Ali Elfilali, Mariama Njie and Jeanna Matthews. 2024. Leveraging Corpus Metadata to Detect Template-based Translation: An Exploratory Case Study of the Egyptian Arabic Wikipedia Edition. Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alshahrani2024leveraging,
  title = {{Leveraging Corpus Metadata to Detect Template-based Translation: An Exploratory Case Study of the Egyptian Arabic Wikipedia Edition}},
  author = {Saied Alshahrani, Hesham Haroon Mohammed, Ali Elfilali, Mariama Njie and Jeanna Matthews},
  booktitle = {Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024},
  year = {2024},
  doi = {10.63317/3gwgj5m5p8zx},
  url = {https://doi.org/10.63317/3gwgj5m5p8zx}
}

References

50 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ameeta Agrawal, Lisa Singh, Elizabeth Jacobs, Yaguang Liu, Gwyneth Dunlevy, Rhitabrat Pokharel, and Varun Uppala. 2023. All Translation Tools Are Not Equal: Investigating the Quality of Language Translation for Forced Migration. In 2023 IEEE 10th International Conference on Data Science and Advanced Analytics (DSAA), pages 1–10. find
  2. Hend Al-Khalifa, Khaloud Al-Khalefah, and Hesham Haroon. 2024. Error Analysis of Pretrained Language Models (PLMs) in English-to-Arabic Machine Translation. Human-Centric Intelligent Systems. find
  3. Asaad Alghamdi, Xinyu Duan, Wei Jiang, Zhenhai Wang, Yimeng Wu, Qingrong Xia, Zhefeng Wang, Yi Zheng, Mehdi Rezagholizadeh, Baoxing Huai, Peilun Cheng, and Abbas Ghaddar. 2023. AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing. arXiv preprint arXiv:2306.06800. ↗
  4. Saied Alshahrani, Norah Alshahrani, Soumyabrata Dey, and Jeanna Matthews. 2023a. Performance Implications of Using Unrepresentative Corpora in Arabic Natural Language Processing. In Proceedings of ArabicNLP 2023, pages 218–231, Singapore (Hybrid). Association for Computational Linguistics. find
  5. Saied Alshahrani, Norah Alshahrani, and Jeanna Matthews. 2023b. DEPTH+: An Enhanced Depth Metric for Wikipedia Corpora Quality. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pages 175–189, Toronto, Canada. Association for Computational Linguistics. find
  6. Saied Alshahrani, Esma Wali, and Jeanna Matthews. 2022. Learning From Arabic Corpora But Not Always From Arabic Speakers: A Case Study of the Arabic Wikipedia Editions. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 361–371, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  7. Maher Asaad Baker. 2022. How I Wrote a Million Wikipedia Articles, 2 edition. BookRix GmbH & Co. KG., Munich, Germany. find
  8. Runa Bhattacharjee and Pau Giner. 2022. You Can Now Use Google Translate to Translate Articles on Wikipedia. Last accessed on 2024-03-01. find
  9. Leo Breiman. 2001. Random Forests. Machine Learning, 45:5–32. find
  10. John Bissell Carroll. 1964. Language and Thought. Prentice-Hall. find
  11. Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: A Library for Support Vector Machines. In ACM Transactions on Intelligent Systems and Technology, volume 2, New York, NY, USA. Association for Computing Machinery. find
  12. John W Chotlos. 1944. IV. A Statistical and Comparative Analysis of Individual Written Language Samples. Psychological Monographs, 56(2):75. find
  13. Noam Cohen. 2008. Open-Source Troubles in Wiki World. The New York Times. Last accessed on 2024-03-01. find
  14. Helmut Daller, Roeland van Hout, and Jeanine Treffers-Daller. 2003. Lexical Richness in the Spontaneous Speech of Bilinguals. Applied Linguistics, 24(2):197–222. find
  15. Alok Das. 2020. Neural Machine Translation (NMT): Inherent Inadequacy, Misrepresentation, and Cultural Bias. International Journal of Translation, 32:115–145. find
  16. M Ester, H P Kriegel, J Sander, and Xu Xiaowei. 1996. A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases With Noise. U.S. Department of Energy Office of Scientific and Technical Information. find
  17. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LI-BLINEAR: A Library for Large Linear Classification. the Journal of Machine Learning Research, 9:1871–1874. find
  18. Pierre Guiraud. 1954. Les Caractères Statistiques du Vocabulaire: Essai de Méthodologie. Presses universitaires de France, Paris, France. find
  19. Pierre Guiraud. 1959. Problèmes et Méthodes de la Statistique Linguistique. D. Reidel, Dordrecht, Holland. find
  20. Ari Hautasaari. 2013. “Could Someone Please Translate This?”: Activity Analysis of Wikipedia Article Translation by Non-experts. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW ’13, page 945–954, New York, NY, USA. Association for Computing Machinery. find
  21. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92–104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. our card
  22. Isaac Johnson and Emily Lescak. 2022. Considerations for Multilingual Wikipedia Research. arXiv preprint arXiv:2204.02483. ↗
  23. Maria Lopez-Medel. 2021. Gender bias in machine translation: an analysis of Google Translate in English and Spanish. Academia.edu. our card
  24. Philip Mccarthy and Scott Jarvis. 2010. MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods, 42:381–92. find
  25. Philip M McCarthy. 2005. An Assessment of the Range and Usefulness of Lexical Diversity Measures and the Potential of the Measure of Textual, Lexical Diversity (MTLD). Ph.D. thesis, The University of Memphis. find
  26. John Mittermeier, Ricardo Correia, Rich Grenyer, Tuuli Toivonen, and Uri Roll. 2021. Using Wikipedia to Measure Public Interest in Biodiversity and Conservation. Conservation Biology, 35. find
  27. El Moatez Billah Nagoudi, Muhammad Abdul-Mageed, AbdelRahim Elmadany, Alcides Alcoba Inciarte, and Md Tawkat Islam Khondaker. 2023. JASMINE: Arabic GPT Models for Few-Shot Learning. arXiv preprint arXiv:2212.10755. ↗
  28. Ranjita Naik, Spencer Rarrick, and Vishal Chowdhary. 2023. Reducing Gender Bias in Machine Translation through Counterfactual Data Generation. arXiv preprint arXiv:2311.16362. ↗ our card
  29. Sergiu Nisioi, Ella Rabinovich, Liviu P. Dinu, and Shuly Wintner. 2016. A Corpus of Native, Non-native and Translated Texts. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4197–4201, Portorož, Slovenia. European Language Resources Association (ELRA). find
  30. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825–2830. find
  31. Marcelo Prates, Pedro Avelar, and Luís Lamb. 2020. Assessing Gender Bias in Machine Translation: A Case Study With Google Translate. Neural Computing and Applications, 32. our card
  32. Motaz Saad and Basem Alijla. 2017. WikiDocsAligner: An Off-the-Shelf Wikipedia Documents Alignment Tool. In Palestinian International Conference on Information and Communication Technology (PICICT). find
  33. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models. arXiv preprint arXiv:2308.16149. ↗ our card
  34. Lucas Shen. 2022. LexicalRichness: A Small Module to Compute Textual Lexical Richness. Last accessed on 2024-03-01. find
  35. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating Gender Bias in Machine Translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics. our card
  36. Mildred C. Templin. 1957. Certain Language Skills in Children: Their Development and Interrelationships, NED–New edition, volume 26. University of Minnesota Press. find
  37. Brian Thompson, Mehak Preet Dhaliwal, Peter Frisch, Tobias Domhan, and Marcello Federico. 2024. A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism. arXiv preprint arXiv:2401.05749. ↗
  38. Stefanie Ullmann and Danielle Saunders. 2021. Google Translate is sexist. What it needs is a little gender-sensitivity training. Last accessed on 2024-03-01. find
  39. Wikidata. 2024a. Wikidata: Requests For Permissions/Bot/DarijaBot. Last accessed on 2024-03-01. find
  40. Wikidata. 2024b. Wikidata: Requests For Permissions/Bot/JarBot. Last accessed on 2024-03-01. find
  41. Wikimedia. 2024. Wikimedia Downloads. Last accessed on 2024-03-01. find
  42. Wikimedia Foundation. 2022. Content Translation – Mediawiki. Last accessed on 2024-03-01. find
  43. Wikimedia Statistics. 2024. New Pages: Egyptian Arabic Wikipedia. Last accessed on 2024-03-01. find
  44. Wikipedia. 2020. Steward Removal of Flags on ARZWiki. Last accessed on 2024-03-01. find
  45. Wikipedia. 2024a. Wiki Markup. Last accessed on 2024-03-01. find
  46. Wikipedia. 2024b. Wikipedia: Bot Policy. Last accessed on 2024-03-01. find
  47. Junjie Wu. 2012. Advances in K-means Clustering: A Data Mining Thinking. Springer Science & Business Media. find
  48. XGBoost. 2024. XGBoost Documentation. Last accessed on 2024-03-01. find
  49. Marie Lisandra Zepeda-Mendoza and Osbaldo Resendis-Antonio. 2013. Hierarchical Agglomerative Clustering, pages 886–887. Springer New York, New York, NY. find
  50. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of LREC 2010 workshop New Challenges for NLP Frameworks, pages 46–50, Valletta, Malta. University of Malta. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alshahrani-2024-leveraging-corpus-metadata-detect-template/