Elamur Research⌕
☰
home / papers / 2024 / Large Language Models as Legal Translators of Arabic Legislation: Do ChatGPT and Gemini Care for Context and Terminology?

Large Language Models as Legal Translators of Arabic Legislation: Do ChatGPT and Gemini Care for Context and Terminology?

Paper · 2024 · Khadija Ait ElFqih and Johanna Monti

Summary

The paper examines whether general-purpose LLMs can translate Arabic legal terminology reliably into English and French. Legal terms are culture-bound and vary by jurisdiction, so translating them requires context. The authors built ARLC, an Arabic legal corpus drawn from legislation, codes, decrees and contracts of several Arab states, and processed it with NooJ. From ARLC they extracted a gold standard of legal terms, each with an example sentence and English and French equivalents. These equivalents were checked against EUR-Lex, IATE and Juremy and then validated by a legal expert and a linguist. ChatGPT-4 and Gemini were prompted zero-shot to translate each term alone (out-of-context) and with its sentence (in-context). Errors were classified with MQM by type and severity. Neither model was dependable. The two models had different strengths depending on language pair and setting. For AR→FR, adding context surprisingly made results worse. The study argues that legal MT for Arabic needs domain terminology, region-aware resources and better context handling, for example through fine-tuning or RAG.

Key points

  • ARLC contains 532,688 tokens of legislation from Morocco, Algeria, Tunisia, UAE, Saudi Arabia, Egypt and Qatar. Sources include official gazettes, ministries of justice and lawyers.
  • The gold dataset has 1,015 Arabic legal terms, each paired with one example sentence plus EN and FR translations. A second annotator added POS tags and geographical usage following ISO 20771:2020.
  • For AR→EN out-of-context, Gemini produced fewer critical errors. In-context, ChatGPT-4 had slightly fewer critical errors but more minor and major ones.
  • For AR→FR, both models did better without context than with it.
  • An example term for an official who certifies marriages was rendered by both models as the adjective "authorized". This changed the part of speech and missed the functional equivalents in the target systems: maire in France, superintendent registrar in England.

Interesting ideas

  • I1. Each term is tested twice, once in isolation and once inside its sentence, which shows whether context actually helps or harms; such a paired ablation exposes hidden reliance on, or confusion from, surrounding text.
  • I2. Errors are counted by category and severity (MQM) instead of with one automatic score, giving actionable diagnostics about what kind of failure occurred and how bad it is.
  • I3. A change of grammatical category in output (a person-noun becoming an adjective) is treated as a distinct error signal, which is cheap to detect automatically.
  • I4. Correct equivalents are defined by institutional function across legal systems rather than by literal wording, so one source concept maps to different officials per jurisdiction.
  • I5. The gold data is validated by two people with complementary expertise, one domain expert for equivalence and one linguist for grammatical and regional annotation, to reduce single-annotator bias.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Run the same item through components with and without its surrounding discourse, treating the bare version as a control that reveals whether context is really being used.
  • I2 (answer): Grade reader and student answers with a typed, severity-weighted error taxonomy so regressions point to a specific failing step.
  • I3 (back): When generating or translating from the meaning graph, verify that each concept keeps its expected category and role, and flag any shift.
  • I4 (world): Store ontology entries for roles and institutions with a function description and jurisdiction or culture tags, so equivalents are chosen by what an entity does.
  • I5 (learn): Split validation of silver data between witnesses that check different aspects, such as structure versus meaning, rather than having one judge everything.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Naples - L'OrientaleNaples · Italy6 works · 456 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksLegal AI · 150 works

Cite

Cite as Khadija Ait ElFqih and Johanna Monti. 2024. Large Language Models as Legal Translators of Arabic Legislation: Do ChatGPT and Gemini Care for Context and Terminology?. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.10

@inproceedings{elfqih2024large,
  title = {{Large Language Models as Legal Translators of Arabic Legislation: Do ChatGPT and Gemini Care for Context and Terminology?}},
  author = {Khadija Ait ElFqih and Johanna Monti},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.10},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.10}
}

References

41 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Almahasees, Z. (2021). Analysing English-Arabic Machine Translation: Google Translate, Microsoft Translator and Sakhr. Routledge. ↗
  2. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N.M., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., García, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A.M., Pillai, T.S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Díaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K.S., Eck, D., Dean, J., Petrov, S., & Fiedel, N. (2022). PaLM: Scaling Language Modeling with Pathways. J. Mach. Learn. Res., 24, 240:1-240:113. ↗
  3. Alves, D., Guerreiro, N., Alves, J., Pombal, J., Rei, R., de Souza, J., Colombo, P., and Martins, A. (2023). Steering Large Language Models for Machine Translation with Fine tuning and In-Context Learning. In Bouamor, H., Pino, J., and Bali, K., editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11127–11148, Singapore. Association for Computational Linguistics. find
  4. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language Models are Few-Shot Learners. ArXiv, abs/2005.14165. ↗
  5. Burchardt, A. (2013). Multidimensional quality metrics: a flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35.
  6. Chang, Y., Wang, X., Wang, J., Wu, Y., Zhu, K., Chen, H., Yang, L., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P.S., Yang, Q., & Xie, X. (2023). A Survey on Evaluation of Large Language Models. ACM Transactions on Intelligent Systems and Technology, 15, 1 - 45. ↗
  7. Darwish, A. (2009). Terminology and translation: A phonological-semantic approach terminology. Writescope Publishers. find
  8. Dai, Z., Yang, Z., Yang, Y., Carbonell, J.G., Le, Q.V., & Salakhutdinov, R. (2019). Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. Annual Meeting of the Association for Computational Linguistics. find
  9. ElFqih, K.A., Di Buono, M.P., Monti, J. (2024). Towards a Linguistic Annotation of Arabic Legal Texts: A Multilingual Electronic Dictionary for Arabic. In: Bartulović, A., Mijić, L., Silberztein, M. (eds) Formalizing Natural Languages: Applications to Natural Language Processing and Digital Humanities. NooJ 2023. Communications in Computer and Information Science, vol 1816. Springer, Cham. https://doi.org/10.1007/978-3-031-56646-2_5 . ↗
  10. ElFqih, K. A., & Monti, J. (2023, September). On the Evaluation of Terminology Translation Errors in NMT and PB-SMT In the Legal Domain: A Study on the Translation of Arabic Legal Documents into English and French. In Proceedings of the Workshop on Computational Terminology in NLP and Translation Studies (ConTeNTS) Incorporating the 16th Workshop on Building and Using Comparable Corpora (BUCC) (pp. 26-35).
  11. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Guo, Q., Wang, M., & Wang, H. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. ArXiv, abs/2312.10997. ↗
  12. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. (2023). Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805,2023. ↗
  13. Winata, G.I., Madotto, A., Lin, Z., Liu, R., Yosinski, J., & Fung, P. (2021). Language Models are Few-shot Multilingual Learners. ArXiv, abs/2109.07684. ↗
  14. Hadi, M. U., Qureshi, R., Shah, A., Irfan, M., Zafar, A., Shaikh, M. B., ... & Mirjalili, S. (2023). A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints. ↗
  15. Hassan, H., Aue, A., Chen, C., Chowdhary, V., Clark, J., Federmann, C., Huang, X., Junczys-Dowmunt, M., Lewis, W.D., Li, M., Liu, S., Liu, T., Luo, R., Menezes, A., Qin, T., Seide, F., Tan, X., Tian, F., Wu, L., Wu, S., Xia, Y., Zhang, D., Zhang, Z., & Zhou, M. (2018). Achieving Human Parity on Automatic Chinese to English News Translation. ArXiv, abs/1803.05567. ↗
  16. Hendy, A., Abdelrehim, M.G., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., Kim, Y., Afify, M., & Awadalla, H.H. (2023). How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. ArXiv, abs/2302.09210. ↗
  17. Herold, C., Rosendahl, J., Vanvinckenroye, J., & Ney, H. (2022). Detecting various types of noise for neural machine translation. Findings of the Association for Computational Linguistics: ACL 2022. http://dx.doi.org/10.18653/v1/2022.findings acl.200 ↗
  18. Huang, Y. (2024). Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies. arXiv preprint arXiv:2402.09282. ↗
  19. Iyer, V., Chen, P., and Birch, A. (2023). Towards Effective Disambiguation for Machine Translation with Large Language Models. In Koehn, P., Haddow, B., Kocmi, T., and Monz, C., editors, Proceedings of the Eighth Conference on Machine Translation, pages482–495, Singapore. Association for Computational Linguistics.
  20. Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguistics.
  21. Gao, J., & Lin, C. (2004). Introduction to the special issue on statistical language modeling. ACM Trans. Asian Lang. Inf. Process., 3, 87-93. ↗
  22. Jiao, W., Huang, J.-T., Wang, W., Wang, X., Shi, S., and Tu, Z. (2023). ParroT: Translating During Chat Using Large Language Models. arXiv preprint arXiv:2304.02426 [cs.CL]. ↗
  23. Killman, J. (2014). Vocabulary accuracy of statistical machine translation in the legal context. Proceedings of the 11th Conference of the Association for Machine Translation in the Americas, 85–98. find
  24. Koehn, P. (2020). Neural machine translation. Cambridge University Press. ↗
  25. Lai, Jinqi & Gan, Wensheng & Wu, Jiayang & Qi, Zhenlian & Yu, Philip. (2023). Large Language Models in Law: A Survey. 10.13140/RG.2.2.15031.09124. ↗
  26. Mediouni, M. (2016). Towards a functional approach to Arabic–english legal translation: The role of comparable/parallel texts. In M. Taibi (Ed.), New Insights into Arabic Translation and Interpreting (pp. 115–160). Multilingual Matters. http://dx.doi.org/10.21832/9781783095254-008 ↗
  27. Mesfar, S. (2008). Analyse morpho-syntaxique automatique et reconnaissance des entités nommées en arabe standard (Doctoral dissertation, Besançon). find
  28. Moslem, Y., Haque, R., & Way, A. (2023). Fine-tuning large language models for adaptive machine translation. arXiv preprint arXiv:2312.12740. ↗
  29. Moslem, Y., Haque, R., Kelleher, J. D., & Way, A. (2023). Adaptive machine translation with large language models. arXiv preprint arXiv:2301.13294. ↗
  30. Rossi, C., & Carre, A. (2022). Machine translation for everyone: Empowering users in the age of artificial intelligence. Language Science Press Berlin, 18, 51. https://doi.org/10.5281/zenodo.6653406 ↗
  31. Sager, J. C. (1990). Practical course in terminology processing. John Benjamins Publishing. find
  32. Schioppa, A., Garcia, X., and Firat, O. (2023). Cross-Lingual Supervision improves Large Language Models Pre-training. arXiv preprint arXiv:2305.11778 [cs.CL]. ↗
  33. Sia, S. and Duh, K. (2022). Prefix Embeddings for In-context Machine Translation. In Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 45–57, Orlando, USA. Association for Machine Translation in the Americas. Silberztein, M.: La formalisation des langues : l’approche de NooJ. find
  34. ISTE Editions, London (2015). find
  35. Siu, S. C. (2023). Chatgpt and GPT-4 for professional translators: Exploring the potential of large language models in translation. Available at SSRN 4448091. ↗
  36. Kombrink, S., Mikolov, T., Karafiát, M., & Burget, L. (2011). Recurrent neural network-based language modeling in meeting recognition. Interspeech. find
  37. Sun, Z. (2023). A short survey of viewing large language models in legal aspect. arXiv preprint arXiv:2303.09136. ↗
  38. Thierry, P. (2022). On" Human Parity" and" Super Human Performance" in Machine Translation Evaluation. Language Resource and Evaluation Conference. find
  39. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language Models are Few-Shot Learners. ArXiv, abs/2005.14165. ↗
  40. Xu, H., Kim, Y. J., Sharaf, A., and Awadalla, H. H. (2023). A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models. arXiv preprintarXiv:2309.11674 [cs.CL]. ↗
  41. Yang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., Yan, D., Yang, F., Deng, F., Wang, F., Liu, F., Ai, G., Dong, G., Zhao, H., Xu, H., Sun, H., Zhang, H., Liu, H., Ji, J., Xie, J., Dai, J., Fang, K., Su, L., Song, L., Liu, L., Ru, L., Ma, L., Wang, M., Liu, M., Lin, M., Nie, N., Guo, P., Sun, R., Zhang, T., Li, T., Li, T., Cheng, W., Chen, W., Zeng, X., Wang, X., Chen, X., Men, X., Yu, X., Pan, X., Shen, Y., Wang, Y., Li, Y., Jiang, Y., Gao, Y., Zhang, Y., Zhou, Z., and Wu, Z. (2023). Baichuan2: Open Large-scale Language Models. arXiv preprint arXiv:2309.10305 [cs.CL]. Gemini Out-of-Context AR→EN Error Severity Multiplier Neutral Minor Major Critical Severity Penalty Multiplier Error Type Weight Error Type Penalty Totals Error Types Dimension Zakraoui, J., Saleh, M., Al-Maadeed, S., & Alja’am, J. M. (2021). Arabic machine translation: A survey with challenges and future directions. IEEE Access, 9,161445–161468. https://doi.org/10.1109/access.2021.3132488 Error Count Terminology 611,050 Style 212,184 Linguistic Convention 102,420 Accuracy 552,625 Audience Appropriateness 609,850 Absolute Penalty Total 2,088.129 *Table 6: MQM Evaluation Scorecard of ChatGPT-4 performance in Out-of-Context AR→EN.* Zhang, X., Rajabi, N., Duh, K., and Koehn, P. (2023). Machine Translation with Large Language Models: Prompting, Few-shot Learning, and Fine-tuning with QLoRA. In Koehn, P., Haddow, B., Kocmi, T., and Monz, C., editors, Proceedings of the Eighth Conference on Machine Translation, pages 468– 481, Singapore. Association for Computational Linguistics Gemini In-Context AR→EN Error Severity Multiplier Neutral Minor Major Critical Severity Penalty Multiplier Error Type Weight Error Type Penalty Totals Error Types Dimension ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elfqih-2024-large-language-models-as-legal/