Elamur Research⌕
☰
home / papers / 2024 / ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models

ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models

Paper · 2024 · Faris Hijazi, Somayah AlHarbi, Abdulaziz AlHussein et al.

Summary

Legal competence of large language models has mostly been tested in English, and Arabic legal knowledge has had no dedicated yardstick. ArabLegalEval fills this gap with a multitask benchmark modeled on MMLU and LegalBench. Its native material comes from publicly available Saudi legal texts published by the Ministry of Justice (MoJ) and the Board of Experts (BoE). The authors scraped these texts with their metadata preserved. From them they synthesized more than 10,000 multiple-choice questions with a strong LLM, comparing three generation strategies and adding automatic validation. They also built an open-ended QA set, NajizQA, from published question–answer material. Finally, legal experts checked Arabic translations of four LegalBench tasks on contracts and privacy policies. Both multilingual and Arabic-centric models were evaluated, including GPT-4, Llama3, Command R Plus and Jais, under several prompting regimes. One-shot prompting usually worked best, and privacy-policy entailment remained the hardest task. The released data and code, plus a generation pipeline that can be reused in other fields such as finance, give Arabic legal AI a common baseline.

Key points

  • Source corpus: the MoJ material comprises 388 circulars along with 67 regulations spanning 5720 subjects. BoE documents cover 448 rules (14,134 subjects). Legal-domain items from ArabicMMLU and public FAQs served as extra seeds.
  • MCQ generation compared three approaches: converting QA into MCQ, Chain of Thought, and retrieval-based in-context learning. Each faced the twin problems of writing the question and producing plausible distractors.
  • NajizQA started from 1358 pairs, which legal experts filtered down to 79. GPT-4 acted as judge of answer similarity, and a "golden model" that returns the ground truth marked the judge's ceiling.
  • The translated LegalBench subset has four tasks:
    • Consumer Contracts QA (400 yes/no items)
    • Contracts QA (88)
    • Privacy Policy QA (10,931)
    • Privacy Policy Entailment (4385)
  • Across the 96 LegalBench experiments, all prompts were in English because that scored higher. Best F1 per task:
    • Consumer contracts: 90%, reached by GPT-4 one-shot and by Llama3 70B with a basic zero-shot prompt.
    • Contracts QA: 99%, Command R Plus few-shot.
    • Privacy Policy Entailment: only 66%, Command R Plus one-shot.

Interesting ideas

  • I1. A judge LLM is calibrated by also scoring a fake model that always outputs the gold answer, which shows the ceiling the judge can actually award.
  • I2. Running QA both with and without the source context showed that the context often hands over the answer outright, so those items test extraction rather than reasoning.
  • I3. Only tasks with few localization requirements were chosen for translation from LegalBench, which keeps the reasoning under test language-neutral while the surface language changes.
  • I4. Benchmark questions were synthesized from raw documents by an LLM and then passed through automatic validation and expert filtering, turning a large unlabeled corpus into a test set.
  • I5. Few-shot examples were held fixed per dataset across all models, so differences in scores reflect the model and not lucky demonstrations.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Before trusting any automatic scorer of Mova's answers, run it on gold outputs as a positive control alongside the negative controls, so scorer bias is separated from model error.
  • I2 (answer): Fable and school-test QA sets could be audited the same way, flagging questions whose answer is a literal span so they are not counted as evidence of inference over the text world.
  • I3 (back): When testing translation through the meta-world, prefer source tasks whose meaning does not depend on culture-specific institutions, so round-trip failures reflect the graph rather than missing local concepts.
  • I4 (learn): LLM teacher tiers could draft questions about texts Mova has parsed, keeping only those that an independent checker confirms against the source, to yield cheap evaluation material for the world stage.
  • I5 (learn): Comparisons between the perceptron parser and gparse, or between repair-rule versions, should freeze prompts and seed examples in the same way so that results stay attributable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Abdullah University of Science and TechnologyJeddah · Saudi Arabia27 works · 1,084 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksLegal AI · 134 works

Cite

Cite as Faris Hijazi, Somayah AlHarbi, Abdulaziz AlHussein, Harethah Abu Shairah, Reem AlZahrani, Hebah AlShamlan, Omar Knio and George Turkiyyah. 2024. ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.20

@inproceedings{hijazi2024arablegaleval,
  title = {{ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models}},
  author = {Faris Hijazi, Somayah AlHarbi, Abdulaziz AlHussein, Harethah Abu Shairah, Reem AlZahrani, Hebah AlShamlan, Omar Knio and George Turkiyyah},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.20},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.20}
}

References

39 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Yousseif Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2024. LAraBench: Benchmarking Arabic AI with Large Language Models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, Malta. Association for Computational Linguistics. ↗
  2. Emad A. Alghamdi, Jezia Zakraoui, and Fares A. Abanmy. 2023. Domain adaptation for arabic machine translation: The case of financial texts. Preprint, arXiv:2309.12863. ↗
  3. Reem Alghamdi, Zhenwen Liang, and Xiangliang Zhang. 2022. ArMATH: a dataset for solving Arabic math word problems. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 351–362, Marseille, France. European Language Resources Association.
  4. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. ↗
  5. Adel Ammar, Anis Koubaa, Bilel Benjdira, Omer Nacar, and Serry Sibaee. 2024. Prediction of arabic legal rulings using large language models. Electronics, 13(4):764. ↗
  6. Anthropic. 2024. Claude 3 haiku: Our fastest model yet. https://www.anthropic.com/news/ claude-3-haiku. Accessed: 2024-06-26. ↗
  7. Cheng-Han Chiang and Hung yi Lee. 2023. Can large language models be an alternative to human evaluations? Preprint, arXiv:2305.01937. ↗
  8. Cohere. 2024a. Command r. https://docs.cohere.com/docs/command-r. Accessed: 2024-06-26. ↗
  9. Cohere. 2024b. Command r+. https://docs.cohere.com/docs/command-r-plus. Accessed: 2024-06-26. ↗
  10. Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali, Majd Hawasly, Nadir Durrani, and Firoj Alam. 2024. LLMeBench: A flexible framework for accelerating llms benchmarking. ↗
  11. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. LawBench: Benchmarking Legal Knowledge of Large Language Models. arXiv. ↗
  12. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023. A framework for few-shot language model evaluation. find
  13. Gregory Goth. 2023. Why are lawyers afraid of AI? Commun. ACM, https://doi.org/10.1145/3631935, 67(1):14–16. ↗
  14. Neel Guha, Julian Nyarko, Daniel E Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. 2023. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. arXiv. ↗
  15. Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, and Tiejun Zhao. 2024. An empirical study of llm-as-ajudge for llm evaluation: Fine-tuned judge models are task-specific classifiers. find
  16. Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Munmun De Choudhury, and Srijan Kumar. 2023. Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries. arXiv. find
  17. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2024. GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(20230254). ↗
  18. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. Dspy: Compiling declarative language model calls into state-of-the-art pipelines. In 12th International Conference on Learning Representations, ICLR 2024, Messe Wien Exhibition and Congress Center, Vienna, Austria, May 7-11, 2024. OpenReview.net. find
  19. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. ↗
  20. Noam Kolt. 2022. Predicting consumer contracts. Berkeley Tech. LJ, 37:71. ↗
  21. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in arabic. Preprint, arXiv:2402.12840. ↗
  22. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315– 7330, Online. Association for Computational Linguistics. ↗
  23. Meta LLaMA. 2024. Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3/. Accessed: 2024-06-26. ↗
  24. Mohammed Maree, Rabee Al-Qasem, and Banan Tantour. 2024. Transforming legal text interactions: leveraging natural language processing and large language models for legal support in palestinian cooperatives. International Journal of Information Technology, 16(1):551–558. ↗
  25. Subhabrata Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Christopher Meek, Ahmed Hassan Awadallah, and Jianfeng Gao. 2021. Few-shot learning evaluation in natural language understanding. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  26. John J. Nay, David Karamardian, Sarah B. Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H. Choi, and Jungo Kasai. 2023. Large language models as tax attorneys: A case study in legal capabilities emergence. arXiv, https://arxiv.org/abs/2306.07075. ↗
  27. OpenAI. 2024a. Gpt-4 technical report. Preprint, arXiv:2303.08774. ↗
  28. Julian Risch, Timo Möller, Julian Gutsch, and Malte Pietsch. 2021. Semantic answer similarity for evaluating question answering models. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 149–157, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  29. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models. arXiv, https://arxiv.org/abs/2308.16149. ↗
  30. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language Models are Multilingual Chain-of-Thought Reasoners. ICLR. ↗
  31. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT — Building open translation services for the World. In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT), Lisbon, Portugal. find
  32. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903. ↗
  33. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.
  34. OpenAI. 2024b. Hello gpt-4o. https://openai.com/ index/hello-gpt-4o/. Accessed: 2024-06-26. ↗
  35. OpenAI. 2024c. New embedding models and api updates. https://openai.com/index/ new-embedding-models-and-api-updates/. Accessed: 2024-07-02. ↗
  36. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc. ↗
  37. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534, Portorož, Slovenia. European Language Resources Association (ELRA).
  38. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. Preprint, arXiv:2404.13076. ↗
  39. Andrew Perlman. 2022. The implications of chatgpt for legal services and society. Suffolk University Law School Research Paper, (22-14). Available at SSRN: http://dx.doi.org/10.2139/ssrn.4294197. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hijazi-2024-arablegaleval-multitask-benchmark-assessing-arabic/