Elamur Research⌕
☰
home / papers / 2023 / Prediction of Arabic Legal Rulings using Large Language Models

Prediction of Arabic Legal Rulings using Large Language Models

Summary

The paper investigates whether current large language models can forecast the outcome of Arabic court cases, a legal-NLP area that has received little attention. The authors use 10,813 real commercial court cases and compare three base models: LLaMA-7b, JAIS-13b and GPT-3.5-turbo. Each model is run zero-shot, one-shot or after task-specific fine-tuning. Some variants first summarize and/or translate the Arabic input into English. This gives 14 configurations.

Outputs were scored by human judges, by GPT-3.5 acting as a grader, and with ROUGE and BLEU. GPT-3.5 configurations were clearly the strongest, with an average score roughly 50% above the Arabic-focused JAIS. LLaMA variants performed poorly. Even the best results remain weak, around 2.4 out of 5 from human raters.

A second finding concerns evaluation. The automatic metrics often disagreed with the human judgments, and this was worst for translated inputs. The study therefore functions both as an early Arabic legal-prediction baseline and as a warning against relying on automatic scores for open-ended legal generation.

Key points

  • Dataset and models: 10,813 real Arabic commercial court cases were tested with LLaMA-7b, JAIS-13b-chat and GPT-3.5-turbo.
  • Variants: Combining training regimes (zero-shot, one-shot, fine-tuning) with input preprocessing (summarization, translation to English) produced 14 configurations.
  • Arabic share of training data: Arabic made up about 29% of JAIS training data. For GPT-3 it was only about 0.03% by words. Despite this, GPT-3.5 variants won by a wide margin.
  • Agreement between human and GPT scores: The correlation was 0.92 for Arabic-input models but only 0.60 for translated-input models.
  • ROUGE precision: ROUGE-1 precision and ROUGE-L precision correlated negatively with human scores (−0.79 and −0.77). The best human score was about 2.4/5.

Interesting ideas

  • I1. Before trusting an automatic metric, compute its correlation with human ratings separately for each pipeline condition; a metric can look acceptable overall yet collapse or even invert on one subgroup. — For Mova (learn): Evaluation should report metric-vs-human agreement per input route (e.g. native vs converted text) as a standing negative control, so a misleading score is caught before it drives training choices.
  • I2. An LLM grader gave its top rating to a configuration that humans judged only moderate, showing that a model-as-judge can reward plausible-sounding output. — For Mova (learn): Answers rated by LLM teacher tiers should be spot-audited against human or rule-checked gold, and tiers whose ratings drift from those audits should be down-weighted.
  • I3. Translating the input before processing seemed to drop or distort key details, which hurt both the predictions and the reliability of their grading. — For Mova (back): When translating through the meta-world, check that the entities and facts in the source graph survive the round trip, rather than judging fluency of the output alone.
  • I4. The strongest LLaMA variant borrowed GPT-3.5's summaries and translations, so its gains came partly from another system; the authors flag that it is not a pure result. — For Mova (learn): Ablations should attribute improvements to the component actually responsible, e.g. silver data from a stronger witness parser versus a change in our own model.
  • I5. The authors plotted input length against human score to see whether long cases are harder. — For Mova (answer): Breaking QA and test accuracy down by text length or depth would show whether failures come from long-range tracking in the world model rather than from local parsing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Law

Cited by

7 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.10260 · arXiv 2310.10260

@misc{ammar2023prediction,
  title = {{Prediction of Arabic Legal Rulings using Large Language Models}},
  author = {Ammar, Adel and Koubaa, Anis and Benjdira, Bilel and Najar, Omar and Sibaee, Serry},
  year = {2023},
  doi = {10.48550/arxiv.2310.10260},
  eprint = {2310.10260},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.10260}
}

References

47 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Habash, N.: Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers, ??? (2010) find
  2. Attia, M.: Handling arabic morphological and syntactic ambiguity within the lfg framework with a view to machine translation. PhD thesis, University of Manchester (2008) find
  3. Dai, A.M., Le, Q.V.: Semi-supervised sequence learning. Advances in neural information processing systems 28 (2015) find
  4. Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146 (2018) ↗
  5. Katz, D.M., Bommarito, M.J., Blackman, J.: A general approach for predicting the behavior of the supreme court of the united states. PloS one 12(4), 0174698 (2017) find
  6. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017) find
  7. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, vol. 27 (2014) find
  8. Almusharraf, M.A.M.I.X.D.N.: Automated and human interaction in written discourse: A contrastive parallel corpus-based investigation of metadiscourse features in machine-human translations. Sage Open (2022) https://doi.org/10.1177/21582440221142210 ↗
  9. Maas, A.L., et al.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (2011) find our card
  10. Khan, A.: Improved multi-lingual sentiment analysis and recognition using deep learning. Journal Of Information Science (2023) https://doi.org/10.1177/01655515221137270 ↗
  11. Chaudhry, H.N., Javed, Y., Kulsoom, F., Mehmood, Z., Khan, Z.I., Shoaib, U., Janjua, S.H.: Sentiment analysis of before and after elections: Twitter data of us election 2020. Electronics 10(17), 2082 (2021) find
  12. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al.: Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 (2022) ↗
  13. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A.A.M., Abid, A., Fisch, A., Brown, A.R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al.: Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 (2022) ↗ our card
  14. Elmadany, A., Nagoudi, E.M.B., Abdul-Mageed, M.: Orca: A challenging benchmark for arabic language understanding. arXiv preprint arXiv:2212.10758 (2022) ↗
  15. Abdelali, A., Mubarak, H., Chowdhury, S.A., Hasanain, M., Mousi, B., Boughorbel, S., Kheir, Y.E., Izham, D., Dalvi, F., Hawasly, M., et al.: Benchmarking arabic ai with large language models. arXiv preprint arXiv:2305.14982 (2023) ↗
  16. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I.: Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356 (2022) ↗ our card
  17. Zhang, Y., Han, W., Qin, J., Wang, Y., Bapna, A., Chen, Z., Chen, N., Li, B., Axelrod, V., Wang, G., et al.: Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037 (2023) ↗
  18. White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., Schmidt, D.C.: A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023) ↗ our card
  19. Zhou, Y., Muresanu, A.I., Han, Z., Paster, K., Pitis, S., Chan, H., Ba, J.: Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910 (2022) ↗ our card
  20. Shin, T., Razeghi, Y., Logan IV, R.L., Wallace, E., Singh, S.: Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 (2020) ↗ our card
  21. Lauderdale, B.E., Clark, T.S.: Scaling politically meaningful dimensions using texts and votes. American Journal of Political Science 58(3), 754–771 (2014) find
  22. Medvedeva, M., Vols, M., Wieling, M.: Using machine learning to predict decisions of the european court of human rights. Artificial Intelligence and Law 27(3), 237–266 (2019) find
  23. AL-Qurishi, M., AlQaseemi, S., Soussi, R.: AraLegal-BERT: A pretrained language model for Arabic Legal text (2022) find
  24. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) find
  25. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023) ↗
  26. Sengupta, N., Sahu, S.K., Jia, B., Katipomu, S., Li, H., Koto, F., Afzal, O.M., Kamboj, S., Pandit, O., Pal, R., et al.: Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149 (2023) ↗ our card
  27. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020) find
  28. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, 27730–27744 (2022) find
  29. Koubaa, A.: Gpt-4 vs. gpt-3.5: A concise showdown. Preprints (2023) https://doi.org/10.20944/preprints202303.0422.v1 ↗
  30. Floridi, L., Chiriatti, M.: Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines 30, 681–694 (2020) find
  31. OpenAI: GPT3 Dataset Language Statistics. https://github.com/openai/gpt-3/tree/master/dataset statistics. Accessed on 09 October 2023 ↗
  32. Shibata, Y., Kida, T., Fukamachi, S., Takeda, M., Shinohara, A., Shinohara, T., Arikawa, S.: Byte pair encoding: A text compression scheme that accelerates pattern matching (1999) find
  33. Bostrom, K., Durrett, G.: Byte pair encoding is suboptimal for language model pretraining. arXiv preprint arXiv:2004.03720 (2020) ↗
  34. Kudo, T., Richardson, J.: Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226 (2018) ↗ our card
  35. Pu, G., Jain, A., Yin, J., Kaplan, R.: Empirical analysis of the strengths and weaknesses of peft techniques for llms. arXiv preprint arXiv:2304.14999 (2023) ↗
  36. Hu, Z., Lan, Y., Wang, L., Xu, W., Lim, E.-P., Lee, R.K.-W., Bing, L., Poria, S.: Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933 (2023) ↗ our card
  37. AGI-Edgerunners: LLM-Adapters Github repository. https://github.com/AGI-Edgerunners/LLM-Adapters. Accessed on 09 October 2023 ↗
  38. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021) ↗
  39. SJP: Saudi Justice Portal. https://sjp.moj.gov.sa. Accessed on 05 October 2023 ↗
  40. PyPI: Selenium Python library. https://pypi.org/project/selenium. Accessed on 05 October 2023 ↗
  41. PyPI: Beautiful Soup Python package. https://pypi.org/project/bs4. Accessed on 04 October 2023 ↗
  42. PyPI: translators Python package. Accessed on 28 September 2023 find
  43. Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318 (2002) find our card
  44. Chen, B., Cherry, C.: A systematic comparison of smoothing techniques for sentence-level bleu. In: Proceedings of the Ninth Workshop on Statistical Machine https://pypi.org/project/translators/. Translation, pp. 362–367 (2014) ↗ our card
  45. NLTK: Bleu Python package. https://www.nltk.org/api/nltk.translate.bleu score.html. Accessed on 04 October 2023 ↗
  46. Lin, C.-Y.: Rouge: A package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004) find
  47. PyPI: rouge Python package. https://pypi.org/project/rouge. Accessed on 04 October 2023 ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ammar-2023-prediction-arabic-legal-rulings-using/