Elamur Research⌕
☰
home / papers / 2025 / Can Large Language Models Predict the Outcome of Judicial Decisions?

Can Large Language Models Predict the Outcome of Judicial Decisions?

Summary

The paper targets Legal Judgment Prediction (LJP) in Arabic, a language with little legal NLP work and almost no public datasets for this task. The authors scraped commercial-court rulings from the Saudi Ministry of Justice publication platform and turned them into structured records. Each record pairs case facts and legal reasoning with the final ruling. From a sampled subset they built training and test splits. To mimic how real users phrase requests, they had GPT-4o write 75 Arabic instructions of differing length and style and spread them evenly over the examples. Two open models, LLaMA-3.2-3B-instruct and LLaMA-3.1-8B-instruct, were tested zero-shot, one-shot and after LoRA fine-tuning. Outputs were scored with overlap and embedding metrics and also graded by an LLM on several quality dimensions. The headline finding is that a small fine-tuned model can approach a larger one on this narrow task at much lower compute cost. This matters for low-resource legal settings, and the authors release the dataset, code and models.

Key points

  • Data source: Saudi Ministry of Justice Judgment Publication Platform (commercial law). The split has 3752 training and 538 test samples, and each input is instruction plus facts and reasons, with the judgment as output.
  • Prompt variety: 75 GPT-4o-generated Arabic instructions are distributed uniformly across data points. For retrieval-based examples, embeddings come from paraphrase-multilingual-MiniLM-L12-v2 (Sentence Transformers).
  • Fine-tuning setup:
    • LoRA with r = 64, α = 64 and dropout 0.1, applied to all attention and MLP projections.
    • AdamW with learning rate 2×10⁻⁴, batch size 32 and max length 2048, on one A100.
    • The text gives inconsistent epoch counts (two vs. one).
  • Automatic metrics are ROUGE-F, BLEU and BERTScore-F. LLaMA-3.1-8B-Instruct also acts as a judge, scoring 1–10 on eight criteria: Accuracy, Relevance, Coherence, Brevity, Legal Language, Faithfulness, Clarity and Consistency.
  • Fine-tuned smaller models perform comparably to larger ones on the task. Dataset, code and models are public on GitHub.

Interesting ideas

  • I1. Training on many reworded versions of the same instruction keeps a model from latching onto one fixed prompt wording, which makes it sturdier against how real people ask. — For Mova (answer): School-course tests and fable QA could be checked against several paraphrased question templates per item, to confirm the reader's answer depends on meaning rather than on surface phrasing.
  • I2. The one-shot demonstration is picked by semantic closeness to the current case, so the in-context example is maximally relevant instead of random. — For Mova (learn): When querying an LLM teacher tier for silver annotations, we could attach the most similar already-verified sentence with its accepted analysis as a guiding example, raising teacher agreement with existing gold.
  • I3. Splitting quality into separate rubric axes (faithfulness, consistency, clarity, and so on) shows how an output fails, which a single overlap number hides. — For Mova (back): The text → graph → text compressor and generator could be graded per axis, especially preservation of facts and absence of contradiction with the meaning graph, so regressions point to a specific defect.
  • I4. The paper scores the same outputs with n-gram overlap, embedding similarity and an LLM grader, since each captures a different notion of closeness. In this setup the grader comes from the same model family as a system under test, which risks self-preference. — For Mova (learn): Our evaluations should pair any LLM judge with negative controls, such as shuffled or corrupted outputs that must score low, to detect a lenient or biased judge.
  • I5. The input is explicitly separated into facts plus reasoning, and the outcome is to be derived from them, rather than treating the case as one undifferentiated blob. — For Mova (world): The text-world could likewise keep asserted states and events apart from the rules or justifications linking them, so an answer can be traced to which fact and which rule produced it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Law

Cited by

2 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 1 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.09768 · arXiv 2501.09768

@misc{kmainasi2025can,
  title = {{Can Large Language Models Predict the Outcome of Judicial Decisions?}},
  author = {Kmainasi, Mohamed Bayan and Shahroor, Ali Ezzat and Al-Ghraibah, Amani},
  year = {2025},
  doi = {10.48550/arxiv.2501.09768},
  eprint = {2501.09768},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.09768}
}

References

51 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al., “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023. LLaMA-3.2-3B-1S LLaMA-3.2-3B-FT LLaMA-3.1-8B-FT 6.94 6.27 7.73 6.42 8.17 8.65 7.44 4.52 3.76 5.18 4.00 4.99 5.14 4.66 6.60 5.87 7.48 6.08 7.90 8.47 7.13 0.53 0.41 0.26 0.76 0.29 0.19 0.11 0.64 0.50 0.39 0.24 0.74 ↗ our card
  2. B. Padiu, R. Iacob, T. Rebedea, and M. Dascalu, “To what extent have llms reshaped the legal domain so far? a scoping literature review,” Information, vol. 15, no. 11, p. 662, 2024. find
  3. M. Liu, T. Okuhara, Z. Dai, W. Huang, H. Okada, F. Emi, and T. Kiuchi, “Performance of advanced large language models (gpt-4o, gpt-4, gemini 1.5 pro, claude 3 opus) on japanese medical licensing examination: A comparative study,” medRxiv, pp. 2024–07, 2024. find
  4. S. Aydin, M. Karabacak, V. Vlachos, and K. Margetis, “Large language models in patient education: a scoping review of applications in medicine,” Frontiers in Medicine, vol. 11, p. 1477898, 2024. find
  5. M. Imran and N. Almusharraf, “Google gemini as a next generation ai educational tool: a review of emerging educational technology,” Smart Learning Environments, vol. 11, no. 1, p. 22, 2024. find
  6. S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, “Large language models: A survey,” arXiv preprint arXiv:2402.06196, 2024. ↗ our card
  7. M. von Schwerin and M. Reichert, “A systematic comparison between open-and closed-source large language models in the context of generating gdpr-compliant data categories for processing activity records,” Future Internet, vol. 16, no. 12, p. 459, 2024. find
  8. A. G. et al., “The llama 3 herd of models,” 2024. find
  9. M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, J. R. Lee, Y. T. Lee, Y. Li, W. Liu, C. C. T. Mendes, A. Nguyen, E. Price, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, X. Wang, R. Ward, Y. Wu, D. Yu, C. Zhang, and Y. Zhang, “Phi-4 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2412.08905 ↗ our card
  10. A. Abdelali, H. Mubarak et al., “Larabench: Benchmarking arabic ai with large language models,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Y. Graham and M. Purver, Eds. St. Julian’s, Malta: Association for Computational Linguistics, Mar 2024, pp. 487– 520. [Online]. Available: https://aclanthology.org/2024.eacl-long.30 ↗ our card
  11. N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C.-M. Chan, W. Chen et al., “Parameter-efficient fine-tuning of large-scale pre-trained language models,” Nature Machine Intelligence, vol. 5, no. 3, pp. 220–235, 2023. find our card
  12. R. Shui, Y. Cao, X. Wang, and T.-S. Chua, “A comprehensive evaluation of large language models on legal judgment prediction,” 2023. [Online]. Available: https://arxiv.org/abs/2310.11761 ↗
  13. D. G. Gordon and T. D. Breaux, “The role of legal expertise in interpretation of legal requirements and definitions,” in 2014 IEEE 22nd International Requirements Engineering Conference (RE). IEEE, 2014, pp. 273–282. find
  14. S. Baez, M. Patiño-Sáenz, J. Martı́nez-Cotrina, D. M. Aponte, J. C. Caicedo, H. Santamarı́a-Garcı́a, D. Pastor, M. L. González-Gadea, M. Haissiner, A. M. Garcı́a et al., “The impact of legal expertise on moral decision-making biases,” Humanities and Social Sciences Communications, vol. 7, no. 1, pp. 1–12, 2020. find
  15. S. T.y.s.s, S. Xu, O. Ichim, and M. Grabmair, “Deconfounding legal judgment prediction for European court of human rights cases towards better alignment with experts,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Y. Goldberg, Z. Kozareva, and Y. Zhang, Eds. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 1120–1138. [Online]. Available: https://aclanthology.org/2022.emnlp-main.74 ↗
  16. K. Darwish, N. Habash, M. Abbas, H. Al-Khalifa, H. T. Al-Natsheh, H. Bouamor, K. Bouzoubaa, V. Cavalli-Sforza, S. R. El-Beltagy, W. El-Hajj et al., “A panoramic survey of natural language processing in the arab world,” Communications of the ACM, vol. 64, no. 4, pp. 72–81, 2021. find
  17. M. B. Kmainasi, R. Khan, A. E. Shahroor, B. Bendou, M. Hasanain, and F. Alam, “Native vs non-native language prompting: A comparative analysis,” arXiv preprint arXiv:2409.07054, 2024. ↗
  18. J. Cui, X. Shen, F. Nie, Z. Wang, J. Wang, and Y. Chen, “A survey on legal judgment prediction: Datasets, metrics, models and challenges,” 2022. [Online]. Available: https://arxiv.org/abs/2204.04859 ↗
  19. F. Kort, “Predicting supreme court decisions mathematically: A quantitative analysis of the “right to counsel” cases,” American Political Science Review, vol. 51, no. 1, pp. 1–12, 1957. find
  20. D. Hsieh, L. Chen, and T. Sun, “Legal judgment prediction based on machine learning: Predicting the discretionary damages of mental suffering in fatal car accident cases,” Applied Sciences, vol. 11, no. 21, p. 10361, 2021. find
  21. I. Chalkidis, I. Androutsopoulos, and N. Aletras, “Neural legal judgment prediction in English,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez, Eds. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 4317–4323. [Online]. Available: https://aclanthology.org/P19-1424/ ↗ our card
  22. R. M. Schmidt, “Recurrent neural networks (rnns): A gentle introduction and overview,” 2019. [Online]. Available: https://arxiv.org/abs/1912.05911 ↗
  23. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023. [Online]. Available: https://arxiv.org/abs/1706.03762 ↗
  24. J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of naacL-HLT, vol. 1. Minneapolis, Minnesota, 2019, p. 2. find
  25. I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and I. Androutsopoulos, “Legal-bert: The muppets straight out of law school,” arXiv preprint arXiv:2010.02559, 2020. ↗ our card
  26. S. Geng, R. Lebret, and K. Aberer, “Legal transformer models may not always help,” arXiv preprint arXiv:2109.06862, 2021. ↗
  27. S. Paul, A. Mandal, P. Goyal, and S. Ghosh, “Pre-trained language models for the legal domain: a case study on indian law,” in Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, 2023, pp. 187–196. find
  28. A. S. Imran, H. Hodnefjeld, Z. Kastrati, N. Fatima, S. M. Daudpota, and M. A. Wani, “Classifying european court of human rights cases using transformer-based techniques,” IEEE Access, vol. 11, pp. 55 664–55 676, 2023. find
  29. A. Maqsood, A. Ul-Hasan, and F. Shafait, “Transformer-based architecture for judgment prediction and explanation in legal proceedings,” in International Workshop on Document Analysis Systems. Springer, 2024, pp. 20–36. find
  30. S. K. Nigam and A. Deroy, “Fact-based court judgment prediction,” in Proceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation, 2023, pp. 78–82. find
  31. V. Malik, R. Sanjay, S. K. Nigam, K. Ghosh, S. K. Guha, A. Bhattacharya, and A. Modi, “ILDC for CJPE: Indian legal documents corpus for court judgment prediction and explanation,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli, Eds. Online: Association for Computational Linguistics, Aug. 2021, pp. 4046–4062. [Online]. Available: https://aclanthology.org/2021.acl-long.313/ ↗ our card
  32. T. Ghosh and S. Kumar, “Evaluating transformer models for legal judgement prediction: A comparative study,” in 2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT). IEEE, 2024, pp. 1–4. find
  33. S. Latisha, S. Favian, and D. Suhartono, “Criminal court judgment prediction system built on modified bert models,” Journal of Advances in Information Technology, vol. 15, no. 2, 2024. find
  34. H. Huang, T. Tang, D. Zhang, W. X. Zhao, T. Song, Y. Xia, and F. Wei, “Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting,” arXiv preprint arXiv:2305.07004, 2023. ↗ our card
  35. X.-P. Nguyen, S. M. Aljunied, S. Joty, and L. Bing, “Democratizing llms for low-resource languages by leveraging their english dominant abilities with linguistically-diverse prompts,” arXiv preprint arXiv:2306.11372, 2023. ↗
  36. M. B. Kmainasi, A. E. Shahroor, M. Hasanain, S. R. Laskar, N. Hassan, and F. Alam, “Llamalens: Specialized multilingual llm for analyzing news and social media content,” arXiv preprint arXiv:2410.15308, 2024. ↗
  37. A. Ammar, A. Koubaa, B. Benjdira, O. Nacar, and S. Sibaee, “Prediction of arabic legal rulings using large language models,” Electronics, vol. 13, no. 4, p. 764, 2024. find our card
  38. S. Vats, A. Zope, S. De, A. Sharma, U. Bhattacharya, S. K. Nigam, S. Guha, K. Rudra, and K. Ghosh, “Llms–the good, the bad or the indispensable?: A use case on legal statute prediction and legal judgment prediction on indian court cases,” in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 12 451–12 474. find
  39. S. K. Nigam, A. Sharma, D. Khanna, N. Shallum, K. Ghosh, and A. Bhattacharya, “Legal judgment reimagined: PredEx and the rise of intelligent AI interpretation in Indian courts,” in Findings of the Association for Computational Linguistics: ACL 2024, L.-W. Ku, A. Martins, and V. Srikumar, Eds. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 4296–4315. [Online]. Available: https://aclanthology.org/2024.findings-acl.255/ ↗
  40. M. B. II and D. M. Katz, “Gpt takes the bar exam,” 2022. [Online]. Available: https://arxiv.org/abs/2212.14402 ↗
  41. I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and I. Androutsopoulos, “Legal-bert: The muppets straight out of law school,” 2020. [Online]. Available: https://arxiv.org/abs/2010.02559 ↗ our card
  42. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” 2023. [Online]. Available: https://arxiv.org/abs/1910.10683 ↗
  43. N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” 2019. [Online]. Available: https://arxiv.org/abs/1908.10084 ↗
  44. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2106.09685 ↗
  45. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2017. [Online]. Available: https://arxiv.org/abs/1412.6980 ↗
  46. C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out, 2004, pp. 74–81. find
  47. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318. ↗ our card
  48. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with bert,” 2020. [Online]. Available: https://arxiv.org/abs/1904.09675 ↗
  49. R. Hada, V. Gumma, A. Wynter, H. Diddee, M. Ahmed, M. Choudhury, K. Bali, and S. Sitaram, “Are large language model-based evaluators the solution to scaling up multilingual evaluation?” in Findings of the Association for Computational Linguistics: EACL 2024, Y. Graham and M. Purver, Eds. St. Julian’s, Malta: Association for Computational Linguistics, Mar. 2024, pp. 1051–1070. [Online]. Available: https://aclanthology.org/2024.findings-eacl.71/ ↗
  50. C.-H. Chiang and H.-y. Lee, “Can large language models be an alternative to human evaluations?” arXiv preprint arXiv:2305.01937, 2023. ↗
  51. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” 2023. [Online]. Available: https://arxiv.org/abs/2309.06180 ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kmainasi-2025-large-language-models-predict-outcome/