Elamur Research⌕
☰
home / papers / 2023 / On the Evaluation of Terminology Translation Errors in NMT and PB-SMT in the Legal Domain: a Study on the Translation of Arabic Legal Documents into English and French

On the Evaluation of Terminology Translation Errors in NMT and PB-SMT in the Legal Domain: a Study on the Translation of Arabic Legal Documents into English and French

Summary

The paper examines how machine translation handles specialised legal terminology when translating Arabic into English and French. Arabic legal vocabulary depends heavily on context and culture, and the authors argue that automatic metrics cannot capture terminology mistakes in this setting. They built a legal corpus of codes, contracts, provisions, constitutions and decrees from six Arab countries. From it they semi-automatically derived a multilingual gold standard of 1015 terms. Each term was tested in isolation and inside a sentence, and translated by an NMT system (GT) and a PB-SMT system (RC). A legal expert validated the reference translations, and the MT errors were annotated manually using a new error typology designed for Arabic legal terms. Overall, NMT made more mistakes than PB-SMT on isolated terms in both language pairs and on in-context terms for AR-EN. For AR-FR, both systems reached the same 94% error rate, but NMT produced more errors of several specific types. The released dataset gives researchers a reference benchmark for an under-studied combination of domain and language.

Key points

  • The gold standard contains 1015 out-of-context and 1015 in-context legal terms with GT and RC outputs, manual error annotations, and expert-validated references for AR-EN and AR-FR. It is published on GitHub (Kaitelfqih/Gold-standard-Terminology-Translation-Evaluation-Data-Set).
  • Source documents come from Morocco, Algeria, Tunisia, the UAE, Saudi Arabia and Egypt.
  • A legal expert fluent in the source and target languages validated the references. An Arabic linguist annotated POS tags and the geographical usage of each term, following ISO 20771:2020.
  • The error typology covers these categories:
    • RE: structural disruption
    • AE: ambiguity
    • CLSRE: concepts absent from the target legal system
    • TDE: term omission
    • LRE: wrong lexical choice
    • GBE: gender-related errors
    • register errors
  • For AR-FR, both systems had a 94% overall error rate, with NMT producing more RE, AE, CLSRE and register errors. Error annotation was done by a single annotator; a second annotator and an inter-annotator agreement study are planned.

Interesting ideas

  • I1. Each term is tested twice, once alone and once inside its sentence, which separates what a system knows about the word from how well it uses context to disambiguate. — For Mova (learn): Evaluation suites could pair isolated-lemma probes with in-sentence probes, so that a failure can be traced to the lexicon or to context handling.
  • I2. The error categories describe the cause of each failure, such as a missing concept, an omitted term, a wrong word choice or a gender clash, rather than giving a single score. — For Mova (back): Generation and translation through the meta-world could log failures in similar cause-based classes, keeping every output error explainable.
  • I3. Both systems turned a noun denoting a person (the marriage officiant) into the adjective "authorized", so the grammatical category changed in translation. — For Mova (back): A check that the source entity type and part of speech survive into the target text would catch this kind of drift automatically.
  • I4. The correct equivalent of the Arabic officiant term is the role-holder in the target legal system ("maire" in France, "superintendent registrar" in England), not a literal gloss. — For Mova (world): Ontology entries could store the function an entity performs, so translation via the meta-world can select the target-culture holder of that function.
  • I5. Each term carries a tag for the region where it is used, because different Arab countries use different words for similar legal practices. — For Mova (world): Region or jurisdiction tags on knowledge-graph lexical entries would let the world model choose the right sense or equivalent for the locale of a document.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation · Law

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{elfqih2023evaluation,
  title = {{On the Evaluation of Terminology Translation Errors in NMT and PB-SMT in the Legal Domain: a Study on the Translation of Arabic Legal Documents into English and French}},
  author = {ElFqih, Khadija Ait and Monti, Johanna},
  booktitle = {Proceedings of the Workshop on Computational Terminology in NLP and Translation Studies (ConTeNTS) Incorporating the 16th Workshop on Building and Using Comparable Corpora (BUCC)},
  year = {2023},
  url = {https://aclanthology.org/2023.contents-1.4/}
}

References

48 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AlMahasees, Z. (2020). Diachronic evaluation of Google Translate, Microsoft Translator, and Sakhr in English-Arabic translation [Master thesis]. Unpublished Master’s Thesis, the University of Western Australia, Australia. find
  2. Alkatheery, E. R. (2023). Google translate errors in legal texts: Machine translation quality assessment. Center for Open Science. http://dx.doi.org/10.31235/osf.io/j4zh7 ↗
  3. Al-Rukban, A., & Saudagar, A. K. J. (2017, December 20). Evaluation of English to Arabic Machine Translation Systems using BLEU and GTM. Proceedings of the 2017 9th International Conference on Education Technology and Computers. http://dx.doi.org/10.1145/3175536.3175570 ↗
  4. Al-Shehab, M. (2013). The translatability of English legal sentences into Arabic by using Google translation. International Journal of English Language and Linguistics Research, 1(3), 18–31. find
  5. Alsohybe, N., Dahan, N., & BaAlwi, F. (2017). Machine-Translation history and evolution: Survey for Arabic-english translations. Current Journal of Applied Science and Technology, 23(4), 1–19. https://doi.org/10.9734/cjast/2017/36124 ↗
  6. Banerjee, S., & Lavie, A. (2005). METEOR: An automatic metric for MT evaluation with improved EESC/COR-FR, d'après le Conseil des communes et régions d’Europe (CCRE), «Gouvernements locaux et régionaux en Europe — Structures et compétences» (2016) (3.5.2022), page 26 Term reference: https://www.citizensadvice.org.uk/family/living-together-marriage-and-civil-partnership/getting-married/correlation with human judgments. Proceedings of the Acl Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 65–72. ↗
  7. Hadla, L., Taghreed, H., & Al-Kabi, M. (2014). Evaluating Arabic to English Machine Translation. International Journal of Advanced Computer Science and Applications, 5, 68–73. https://doi.org/10.14569/IJACSA.2014.051112#sth ash.NW2l6vl5.dpuf Baruah, R., & Singh, A. K. (2023). A clinical practice by machine translation on low resource languages. In Natural Language Processing in Healthcare (pp. 1–17). CRC Press. http://dx.doi.org/10.1201/9781003138013-1 ↗
  8. Halimi, S. A. (2017). Contextualizing translation decisions in legal system-bound and international multilingual contexts. Between Specialised Texts and Institutional Contexts – Competence and Choice in Legal Translation, 3(1), 20–46. https://doi.org/10.1075/ttmc.3.1.03hal ↗
  9. Berrichi, S., & Mazroui, A. (2021). Addressing limited vocabulary and long sentences constraints in english–arabic neural machine translation. Arabian Journal for Science and Engineering, 46(9), 8245– 8259. https://doi.org/10.1007/s13369-020-05328-2 ↗
  10. Han, L. (2016). Machine translation evaluation resources and methods: A survey. arXiv Preprint arXiv:1605.04515. ↗
  11. Bouamor, H., Alshikhabobakr, H., Mohit, B., & Of lazer, K. (2014). A human judgement corpus and a metric for Arabic MT evaluation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). http://dx.doi.org/10.3115/v1/d14-1026 ↗
  12. Haque, R., Hasanuzzaman, M., & Way, A. (2019). TermEval: An automatic metric for evaluating terminology translation in MT. Springer. find
  13. Haque, R., Hasanuzzaman, M., & Way, A. (2020). Analysing terminology translation errors in statistical and neural machine translation. Machine Translation, 34(2–3), 149–195. https://doi.org/10.1007/s10590-020-09251-z ↗
  14. Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014). On the properties of neural machine translation: Encoder–Decoder approaches. Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. http://dx.doi.org/10.3115/v1/w14-4012 ↗ our card
  15. Herold, C., Rosendahl, J., Vanvinckenroye, J., & Ney, H. (2022). Detecting various types of noise for neural machine translation. Findings of the Association for Computational Linguistics: ACL 2022. http://dx.doi.org/10.18653/v1/2022.findings acl.200 ↗
  16. Cuong, H., & Sima’an, K. (2017). A survey of domain adaptation for statistical machine translation. Machine Translation, 31(4), 187–224. https://doi.org/10.1007/s10590-018-9216-8 ↗
  17. Izwaini, S. (2006). Problems of Arabic machine translation: evaluation of three systems. Proceedings of the International Conference on the Challenge of Arabic for NLP/MT, 118–148. https://aclanthology.org/2006.bcs-1.11 ↗
  18. Darwish, A. (2009). Terminology and translation: A phonological-semantic approach to Arabic terminology. Writescope Publishers. find
  19. Dugonik, J., Sepesy Maučec, M., Verber, D., & Brest, J. (2023). Reduction of neural machine translation failures by incorporating statistical machine translation. Mathematics, 11(11), 2484. https://doi.org/10.3390/math11112484 ↗
  20. Junczys-Dowmunt, M., Dwojak, T., & Hoang, H. (2016). Is neural machine translation ready for deployment? A case study on 30 translation directions. arXiv Preprint arXiv:1610.01108. ↗ our card
  21. El Marouani, M., Boudaa, T., & Enneya, N. (2020). Statistical error analysis of machine translation: The case of Arabic. Computación y Sistemas, 24(3). https://doi.org/10.13053/cys-24-3-3289 ↗
  22. Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014). A convolutional neural network for modelling sentences. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). http://dx.doi.org/10.3115/v1/p14-1062 ElFqih, K. A., di Buono, M. P., & Monti, J. Towards a Linguistic Annotation of Arabic Legal Texts: A Multilingual Electronic Dictionary for Arabic. In Book of Abstracts (p. 17). ↗ our card
  23. Khazin, K. M., Sanjaya, D., Siregar, M., Meisuri, & Adisaputra, A. (2023). Comparison of machine translations (MT) technology; statistical (SMT) vs. neural (NMT). ADVANCES IN FRACTURE AND DAMAGE MECHANICS XX. http://dx.doi.org/10.1063/5.0133311 ↗
  24. Gamal, D., Alfonse, M., Jimenez-Zafra, S. M., & Aref, M. (2022, May 8). Survey of Arabic machine translation, methodologies, progress, and challenges. 2022 2nd International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC). http://dx.doi.org/10.1109/miucc55081.2022.97817 76 ↗
  25. Killman, J. (2014). Vocabulary accuracy of statistical machine translation in the legal context. Proceedings of the 11th Conference of the Association for Machine Translation in the Americas, 85–98. find
  26. Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2022). A survey of evaluation metrics used for NLG systems. ACM Computing Surveys, 55(2), 1– 39. https://doi.org/10.1145/3485766 ↗
  27. Koehn, P. (2020). Neural machine translation. Cambridge University Press. find
  28. Sepesy Maučec, M., & Donaj, G. (2019). Machine translation and the evaluation of its quality. In A. Sadollah & S. Tilendra (Eds.), Recent Trends in Computational Intelligence. IntechOpen. http://dx.doi.org/10.5772/intechopen.89063 ↗
  29. Koehn, P., & Knowles, R. (2017). Six challenges for neural machine translation. Proceedings of the First Workshop on Neural Machine Translation. http://dx.doi.org/10.18653/v1/w17-3204 ↗
  30. Koehn, P., Och, F. J., & Marcu, D. (2003). Statistical phrase-based translation. Defense Technical Information Center. http://dx.doi.org/10.21236/ada461156 ↗
  31. Thierry, P. (2022). On" Human Parity" and" Super Human Performance" in Machine Translation Evaluation. Language Resource and Evaluation Conference. find
  32. Lavie, A., & Denkowski, M. J. (2009). The Meteor metric for automatic evaluation of machine translation. Machine Translation, 23(2–3), 105–115. https://doi.org/10.1007/s10590-009-9059-4 ↗
  33. Tiedemann, J. (2012). Parallel data, tools and interfaces in OPUS. find
  34. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017, June 12). Attention is all you need. arXiv.Org. https://arxiv.org/abs/1706.03762 ↗
  35. Lee, S., Lee, J., Moon, H., Park, C., Seo, J., Eo, S., Koo, S., & Lim, H. (2023). A Survey on Evaluation Metrics for Machine Translation. Mathematics, 11(4), 1006. find
  36. Vintar, Š. (2018). Terminology translation accuracy in statistical versus neural MT: An evaluation for the English-Slovene language pair. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 34–37. find
  37. Madi, N., & Al-Khalifa, H. (2020). Error detection for Arabic text using neural sequence labeling. Applied Sciences, 10 (15), 5279. https://doi.org/10.3390/app10155279 ↗
  38. Wang, W., Tian, Y., Ngiam, J., Yang, Y., Caswell, I., & Parekh, Z. (2020). Learning a multi-domain curriculum for neural machine translation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. http://dx.doi.org/10.18653/v1/2020.acl-main.689 ↗
  39. Mediouni, M. (2016). Towards a functional approach to Arabic–english legal translation: The role of comparable/parallel texts. In M. Taibi (Ed.), New Insights into Arabic Translation and Interpreting (pp. 115–160). Multilingual Matters. http://dx.doi.org/10.21832/9781783095254-008 ↗
  40. Silberztein, M. (2015). La formalisation des langues: l'approche de NooJ. ISTE Group. find
  41. Müller, M., Rios, A., & Rico, S. (2019). Domain robustness in neural machine translation. arXiv Preprint arXiv:1911.03109. ↗ our card
  42. Watson, D., Zalmout, N., & Habash, N. (2018). Utilizing character and word embeddings for text normalization with sequence-to-sequence models. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. http://dx.doi.org/10.18653/v1/d18-1097 ↗
  43. Nishimura, T., & Akiba, T. (2017, August). Addressing unknown word problem for neural machine translation using distributee representations of words as input features. 2017 International Conference on Advanced Informatics, Concepts, Theory, and Applications (ICAICTA). http://dx.doi.org/10.1109/icaicta.2017.8090977 ↗
  44. Zakraoui, J., Saleh, M., Al-Maadeed, S., & AlJa’am, J. M. (2020, April). Evaluation of Arabic to English machine translation systems. 2020 11th International Conference on Information and Communication Systems (ICICS). http://dx.doi.org/10.1109/icics49469.2020.239518 ↗
  45. Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002). Bleu: a method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics -ACL ’02. http://dx.doi.org/10.3115/1073083.1073135 ↗ our card
  46. Zakraoui, J., Saleh, M., Al-Maadeed, S., & Alja’am, J. M. (2021). Arabic machine translation: A survey with challenges and future directions. IEEE Access, 9, 161445–161468. https://doi.org/10.1109/access.2021.3132488 ↗
  47. Rossi, C., & Carre, A. (2022). Machine translation for everyone: Empowering users in the age of artificial intelligence. Language Science Press Berlin, 18, 51. https://doi.org/10.5281/zenodo.6653406 ↗
  48. Ziemski, M., Junczys-Dowmunt, M., & Pouliquen, B. (2016). The United Nations parallel corpus v1. 0. Proceedings of the Tenth International Conference Sager, J. C. (1990). Practical course in terminology processing. John Benjamins Publishing. on Language Resources and Evaluation (LREC’16), 3530–3534. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elfqih-2023-evaluation-terminology-translation-errors-nmt/