Elamur Research⌕
☰
home / papers / 2017 / A Parallel Corpus for Evaluating Machine Translation between Arabic and European Languages

A Parallel Corpus for Evaluating Machine Translation between Arabic and European Languages

Paper · 2017 · Nizar Habash, Nasser Zalmout, Dima Taji et al.

Summary

Most Arabic machine translation resources pair Arabic with English, and only a handful pair it with other languages. As a result, there is almost no shared test material for translation between Arabic and most European languages. The authors address this by building Arab-Acquis. They take the tuning, development and test portions of the JRC-Acquis corpus, which already exist in 22 EU languages, and have professional translators render them into Standard Arabic. Each sentence gets two independent Arabic versions: one made from the English text and one from the French. The resource covers about 12,000 sentences and roughly 687K Arabic words, and it keeps the data splits of Koehn et al. (2009).

To give first benchmarks, the authors chain Moses phrase-based systems through English in both directions between Arabic and every JRC-Acquis language. Pivoted scores are far below direct English scores, which shows how much room there is for better Arabic–European MT. Because every language uses identical content, cross-language comparisons are not confounded by differences in the data.

Key points

  • Data: Arab-Acquis contains over 12,000 JRC-Acquis sentences, each translated twice into Arabic: once from English (~340K words) and once from French (~347K words). The total is 687,344 translated words, and the splits are tune, dev (4,108 sentences) and test (4,107 sentences).
  • Translation process: three companies did the work, one each from Egypt, Lebanon and Jordan. Files were assigned so that no company received the same file in both source languages. Translation took 44–90 days, and the total cost was just over USD 43,200 (about $0.056/word from English and $0.073/word from French).
  • Systems: the authors built 21 En→X and 21 X→En Moses systems, which replicate Koehn et al. (2009). They added Ar↔En systems trained on the UN Corpus and evaluated everything with case-insensitive BLEU and a 5-gram LM. Pivoting was simple chaining through English.
  • Results: En→X BLEU is roughly double the pivoted Ar→En→X BLEU. French scores best into X, Maltese scores best from X (72.3 X→En), and Hungarian and Estonian score worst. Direct and pivoted scores correlate strongly (r = 0.97 for X→En vs. X→En→Ar, r = 0.93 for En→X vs. Ar→En→X).
  • Predictors of difficulty: average words per sentence in X tracks En→X BLEU (r = 0.82) and Ar→En→X BLEU (r = 0.91), but it predicts poorly when X is the source. Words per sentence is itself almost a mirror of type-token ratio (r = −0.96). Rankings of language families also align with into-English and into-Arabic scores.

Interesting ideas

  • I1. Creating each pair of reference translations from two different source languages builds natural wording variety into the references, which makes multi-reference scoring fairer than two translations of a single source.
  • I2. Reusing one fixed set of content across all 22 languages means that score differences reflect the language itself, not the sampled data.
  • I3. The average number of words per sentence on parallel text serves as a cheap, surprisingly strong stand-in for morphological richness and predicts how hard a target is.
  • I4. The authors deliberately extend existing community data splits instead of inventing new ones, so new results stay directly comparable with earlier work.
  • I5. Assigning files so that no single translator handles both versions keeps the two references independent and avoids correlated mistakes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): when scoring the text → graph → text compressor or meta-world translation, collect references derived from differently phrased inputs, so a valid paraphrase is not penalised as an error.
  • I2 (learn): build evaluation suites where the same underlying story or facts appear in several surface forms, so that drops in accuracy can be attributed to syntax or wording rather than content.
  • I3 (tokens): track tokens per sentence and type-token ratio on incoming genres as a quick signal of where the tagger and lemmatizer will struggle.
  • I4 (tree): when adding or converting treebanks, keep the official UD train/dev/test boundaries so that LAS numbers for the perceptron parser and gparse remain comparable with published scores.
  • I5 (learn): in silver-data agreement, make sure witness parsers or LLM teacher tiers are truly independent (different models and inputs), so that their agreement means something.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksSemitic languages · 2,908 works

Cite

Cite as Nizar Habash, Nasser Zalmout, Dima Taji, Hieu Hoang and Maverick Alzate. 2017. A Parallel Corpus for Evaluating Machine Translation between Arabic and European Languages. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{habash2017parallel,
  title = {{A Parallel Corpus for Evaluating Machine Translation between Arabic and European Languages}},
  author = {Nizar Habash, Nasser Zalmout, Dima Taji, Hieu Hoang and Maverick Alzate},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers},
  year = {2017},
  doi = {10.18653/v1/e17-2038},
  url = {https://doi.org/10.18653/v1/e17-2038}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nizar Habash. 2010. Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. ↗
  2. Hassan Al-Haj and Alon Lavie. 2012. The impact of arabic morphological segmentation on broad-coverage english-to-arabic statistical machine translation. Machine translation, 26(1-2):3–24. ↗
  3. Olivier Hamon and Khalid Choukri. 2011. Evaluation methodology and results for english-to-arabic mt. In Proceedings of the 13th Machine Translation Summit (MT Summit XIII), pages 480–487, Xiamen, China. find
  4. Alexandra Birch, Miles Osborne, and Philipp Koehn. 2008. Predicting success in machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 745–754, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  5. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2016. Google’s multilingual neural machine translation system: Enabling zero-shot translation. arXiv preprint arXiv:1611.04558. ↗
  6. Mohamed Mahdi Boudabous, Nouha Chaâben Kammoun, Nacef Khedher, Lamia Hadrich Belguith, and Fatiha Sadat. 2013. Arabic wordnet semantic relations enrichment through morpho-lexical patterns. In Communications, Signal Processing, and their Applications (ICCSPA), 2013 1st International Conference on, pages 1–6, Sharjah, United Arab Emirates. IEEE. find
  7. Philipp Koehn and Barry Haddow. 2009. Interactive assistance to human translators using statistical machine translation methods. In Proceedings of the 12th Machine Translation Summit (MT XII), pages 1–8, Ottawa, Canada. find
  8. Philipp Koehn, Hieu Hoang, Alexandra Birch, Christopher Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Christopher Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177–180, Prague, Czech Republic. Association for Computational Linguistics. find
  9. Mauro Cettolo, Nicola Bertoldi, Marcello Federico, and FBK-Fondazione Bruno Kessler. 2011. Bootstrapping Arabic-Italian SMT through comparable texts and pivot translation. In Proceedings of the 15th Conference of the European Association for Machine Translation (EAMT), pages 249–256, Leuven, Belgium. find
  10. Ahmed El Kholy and Nizar Habash. 2010. Techniques for Arabic Morphological Detokenization and Orthographic Denormalization. In Proceedings of the seventh International Conference on Language Resources and Evaluation (LREC), pages 45–51, Valletta, Malta. find
  11. Philipp Koehn, Alexandra Birch, and Ralf Steinberger. 2009. 462 Machine Translation Systems for Europe. In Proceedings of the 12th Machine Translation Summit (MT XII), pages 65–72, Ottawa, Canada. find
  12. Ahmed El Kholy and Nizar Habash. 2012. Orthographic and morphological processing for English– Arabic statistical machine translation. Machine Translation, 26(1-2):25–45. ↗
  13. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Conference Proceedings: the tenth Machine Translation Summit, pages 79–86, Phuket, Thailand. AAMT. find
  14. Ahmed El Kholy, Nizar Habash, Gregor Leusch, Evgeny Matusov, and Hassan Sawaf. 2013. Language independent connectivity strength features for phrase pivot statistical machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, volume 2, pages 412–418, Sofia, Bulgaria. Association for Computational Linguistics. find
  15. Joseph Olive, Caitlin Christianson, and John McCary. 2011. Handbook of natural language processing and machine translation: DARPA global autonomous language exploitation. Springer Science & Business Media, Verlag New York. ↗
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, PA, USA. Association for Computational Linguistics.
  17. European Parliament. 2016. Multilingualism in the european parliament. http://www.europarl.europa.eu/aboutparliament/en/ 20150201PVL00013/Multilingualism. Accessed: 2016-07-15. ↗
  18. Nizar Habash and Jun Hu. 2009. Improving arabic-chinese statistical machine translation using english as pivot language. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 173–181, Athens, Greece. Association for Computational Linguistics. find
  19. Arfath Pasha, Mohamed Al-Badrashiny, Ahmed El Kholy, Ramy Eskander, Mona Diab, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In In Proceedings of LREC, pages 1094– 1101, Reykjavik, Iceland.
  20. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods, pages 15–22. Springer, Netherlands. ↗
  21. Alexandre Rafalovitch, Robert Dale, et al. 2009. United nations general assembly resolutions: A six-language parallel corpus. In Proceedings of the 12th Machine Translation Summit (MT XII), volume 12, pages 292–299, Ottawa, Canada. find
  22. Reshef Shilon, Nizar Habash, Alon Lavie, and Shuly Wintner. 2012. Machine translation between hebrew and arabic. Machine translation, 26(1-2):177– 195. ↗
  23. Michel Simard, George F Foster, and Pierre Isabelle. 1993. Using cognates to align sentences in bilingual corpora. In Proceedings of the 1993 conference of the Centre for Advanced Studies on Collaborative research: distributed computing-Volume 2, pages 1071–1082, Toronto, Ontario, Canada. IBM Press.
  24. Ralf Steinberger, Bruno Pouliquen, Anna Widiger, Camelia Ignat, Tomaz Erjavec, Dan Tufis, and Dániel Varga. 2006. The JRC-Acquis: A multilingual aligned parallel corpus with 20+ languages. arXiv preprint cs/0609058.
  25. Masao Utiyama and Hitoshi Isahara. 2007. A comparison of pivot methods for phrase-based statistical machine translation. In HLT-NAACL, pages 484– 491, Rochester, NY, USA. Association for Computational Linguistics.
  26. Vered Volansky, Noam Ordan, and Shuly Wintner. 2015. On the features of translationese. Digital Scholarship in the Humanities, 30(1):98–118. find
  27. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F. Zaidan, and Chris Callison-Burch. 2012. Machine Translation of Arabic Dialects. In Proceedings of NAACL-HLT, pages 49– 59, Montréal, Canada. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/habash-2017-parallel-corpus-evaluating-machine-translation/