Elamur Research⌕
☰
home / papers / 2021 / Investigating Code-Mixed Modern Standard Arabic-Egyptian to English Machine Translation

Investigating Code-Mixed Modern Standard Arabic-Egyptian to English Machine Translation

Paper · 2021 · El Moatez Billah Nagoudi, AbdelRahim Elmadany and Muhammad Abdul-Mageed

Summary

This paper asks how well modern neural MT handles a source side that mixes Modern Standard Arabic with Egyptian Arabic (MSAEA) when translating into English. The authors built their system as an entry in the unsupervised track of a code-switching MT shared task, in which participants received only unlabeled test data. The authors gathered about 61M MSA-English pairs from OPUS. They filtered these with a cross-lingual similarity model and a trigram overlap check, which left roughly 55M pairs. They also assembled about 257K dialectal-English pairs covering Egyptian, Levantine and Gulf Arabic.

They compared two families of systems:

  • a Transformer trained from scratch on MSA only, which is a zero-shot setting for Egyptian;
  • pre-trained mT5 and mBART fine-tuned on the dialectal data.

The from-scratch model already scored reasonably well, which the authors attribute to lexical overlap between MSA and Egyptian. Adding dialect data helped every system. The best model, the from-scratch Transformer fine-tuned on dialects with truecased output, reached 25.72 BLEU and ranked first. The study shows that a standard variety plus related dialects can stand in for scarce code-mixed training data.

Key points

  • The authors cleaned the OPUS MSA-English data in two steps. First, they kept only pairs with a cross-lingual similarity score (Yang et al., 2019 model) between 0.30 and 0.99. Second, they ran a word-trigram sliding-window overlap check (Barrón-Cedeño & Rosso) with a >75% threshold. This reduced 61M pairs to about 55.2M; 10K were held out as MSA-Dev.
  • The dialectal data came from Zbib et al. (2012), MADAR and the Qatari-English speech corpus. It contains 56K Egyptian, 160K Levantine and 40.7K Gulf pairs; Maghrebi was excluded. DA-Train is about 250.7K pairs, and DA-Dev is 6K Egyptian pairs.
  • Test results for the from-scratch Transformer (S2ST) were:
    • 21.34 BLEU in the zero-shot EA setting;
    • 22.51 BLEU after dialect fine-tuning;
    • 25.72 BLEU after dialect fine-tuning with truecased output.
  • Fine-tuned mT5 and mBART stayed below 20 BLEU on Test, even with truecasing. However, they handled Egyptian-specific tokens well. Truecasing added roughly 2.5 BLEU on average.
  • Scores were higher on Test than on Dev. A binary MSA-DA classifier supported the authors' explanation: the test set contains proportionally more MSA than the dev set.

Interesting ideas

  • I1. The similarity filter keeps pairs inside a band rather than above a floor: it drops pairs scoring near 1 (copies of each other) as well as low-scoring pairs (bad translations). This catches untranslated copies that a plain minimum threshold would let through.
  • I2. When dev and test scores diverge, the authors run a variety classifier over each split to measure how much dialect it contains, instead of guessing at the cause. Profiling the composition of each split makes gaps between evaluation sets explainable.
  • I3. The authors train only on the standard variety and evaluate on mixed text as an explicit zero-shot baseline. This shows how much transfer comes from shared vocabulary alone, before any in-domain data is added.
  • I4. Pooling several related dialects for fine-tuning helped the single target dialect, even though most of the added data was not Egyptian. Neighbouring varieties can supply useful signal when the exact target is scarce.
  • I5. Light preprocessing replaces URLs, user mentions and hashtags with generic placeholder tokens and keeps the rest of the text faithful. This reduces vocabulary noise without distorting natural language.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When building silver data, discard both suspiciously perfect agreement (e.g., a sentence copied unchanged) and weak agreement between witnesses, so that trivial or leaked examples do not inflate training.
  • I2 (learn): Before comparing parser scores across test sets, tag each set by genre or register, so that a score jump can be traced to a shift in the data rather than to the model.
  • I3 (tree): Train the parser on standard-English treebanks only, then measure it on informal or dialectal English, to quantify how far plain transfer goes before adding converted or dialect treebanks.
  • I4 (tree): When gold data for a niche register (e.g., social media) is small, mix in treebanks from adjacent registers and track the gain on the target register.
  • I5 (tokens): Have the tokenizer map URLs, handles and hashtags to typed placeholder tokens with fixed tags, so the tagger and parser treat them consistently and the original strings stay recoverable for explanation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Machine translation · 1,087 worksSemitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as El Moatez Billah Nagoudi, AbdelRahim Elmadany and Muhammad Abdul-Mageed. 2021. Investigating Code-Mixed Modern Standard Arabic-Egyptian to English Machine Translation. Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.calcs-1.8

@inproceedings{nagoudi2021investigating,
  title = {{Investigating Code-Mixed Modern Standard Arabic-Egyptian to English Machine Translation}},
  author = {El Moatez Billah Nagoudi, AbdelRahim Elmadany and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching},
  year = {2021},
  doi = {10.18653/v1/2021.calcs-1.8},
  url = {https://doi.org/10.18653/v1/2021.calcs-1.8}
}

References

26 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic. arXiv preprint arXiv:2101.01785. ↗
  2. John J. Gumperz. 1982. Discourse Strategies. Studies in Interactional Sociolinguistics. Cambridge University Press. find
  3. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  4. Muhammad Abdul-Mageed, AbdelRahim Elmadany, Dinesh Pabbi, Kunal Verma, Rannie Lin, et al. 2021. Mega-cov: A billion-scale dataset of 100+ languages for covid-19. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3402–3420. ↗
  5. Philipp Koehn, Marcello Federico, Wade Shen, Nicola Bertoldi, Ondrej Bojar, Chris Callison-Burch, Brooke Cowan, Chris Dyer, Hieu Hoang, Richard Zens, et al. 2007. Open source toolkit for statistical machine translation: Factored translation models and confusion network decoding. In Final Report of the Johns Hopkins 2006 Summer Workshop. https://code-switching.github.io/2021 62 7 ↗
  6. Hassan Sajjad, Kareem Darwish, and Yonatan Belinkov. 2013. Translating dialectal arabic to english. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–6. find
  7. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. arXiv preprint arXiv:1804.07755. ↗
  8. Lucian Vlad Lita, Abe Ittycheriah, Salim Roukos, and Nanda Kambhatla. 2003. Truecasing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 152–159. find
  9. Wael Salloum, Heba Elfardy, Linda Alamir-Salloum, Nizar Habash, and Mona Diab. 2014. Sentence level dialect identification for machine translation system selection. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 772–778. find
  10. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. ↗
  11. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. ↗
  12. Karima Meftouh, Salima Harrat, Salma Jamoussi, Mourad Abbas, and Kamel Smaili. 2015. Machine translation experiments on PADIC: A parallel Arabic DIalect corpus. In Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation, pages 26–34, Shanghai, China.
  13. Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Krishna Rallabandi, and Alan W. Black. 2019. A survey of code-switched speech and language processing. CoRR, abs/1904.00784. ↗
  14. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. 2012:2214–2218. find
  15. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038. ↗
  16. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  17. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. arXiv preprint arXiv:1806.00187. ↗
  18. Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2019. Code-switched language models using neural based synthetic data from parallel sentences. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 271–280, Hong Kong, China. Association for Computational Linguistics. find
  19. Mai Oudah, Amjad Almahairi, and Nizar Habash. 2019. The impact of preprocessing on arabic-english statistical and neural machine translation. arXiv preprint arXiv:1906.11751. ↗
  20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. ↗
  21. Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pier-ric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45.
  22. Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018. Language modeling for code-mixing: The role of linguistic theory based synthetic data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1543–1553, Melbourne, Australia. Association for Computational Linguistics.
  23. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mT5: A massively multilingual pre-trained text-to-text transformer. ↗
  24. Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi. 2020a. Arabench: Benchmarking dialectal arabic-english machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5094–5107. find
  25. Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et al. 2019. Multilingual universal sentence encoder for semantic retrieval. arXiv preprint arXiv:1907.04307. ↗
  26. Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi. 2020b. AraBench: Benchmarking dialectal Arabic-English machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5094–5107, Barcelona, Spain (Online). International Committee on Computational Linguistics. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar Zaidan, and Chris Callison-Burch. 63 8 2012. Machine translation of arabic dialects. In Proceedings of the 2012 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 49– 59. 64 9

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nagoudi-2021-investigating-code-mixed-modern-standard/