Elamur Research⌕
☰
home / papers / 2023 / ALJP: An Arabic Legal Judgment Prediction in Personal Status Cases Using Machine Learning Models

ALJP: An Arabic Legal Judgment Prediction in Personal Status Cases Using Machine Learning Models

Summary

The paper addresses Legal Judgment Prediction (LJP) for Arabic, a language the authors say had no prior LJP work. It focuses on Saudi personal status law, specifically custody and annulment of marriage. The authors built a new dataset by combining sample cases published by the Saudi Ministry of Justice with cases simulated by legal experts. They then framed prediction as multi-class classification with two tasks. The first maps the pleading text to a judgment and to the supporting law articles or evidence. The second maps the plaintiff's claim and the defendant's answer to probable outcomes. They compared SVM, LR, LSTM and BiLSTM, each with TF-IDF and word2vec features. For custody judgments, SVM with word2vec reached 88% accuracy, and for annulment, LR with TF-IDF reached 78%. Predicting law articles proved much harder, peaking at 25% for custody and 50% for annulment. Classical models matched or beat the neural ones, which the authors attribute to the small dataset. The work matters mainly as a first Arabic resource and baseline for this task.

Key points

  • Domain: Saudi personal status cases (custody, annulment of marriage). Data comes from Ministry of Justice sample collections plus expert-simulated cases.
  • The study has two tasks:
    • pleading → judgment and law article/evidence;
    • claim + defense answer → most probable outcome.
  • Eight configurations were tested: SVM, LR, LSTM and BiLSTM, each crossed with TF-IDF and word2vec. Metrics were accuracy, precision, recall and F1.
  • Judgment prediction:
    • custody: 88% with SVM-word2vec;
    • annulment: 78% with LR-TFIDF, and SVM-TFIDF also hit 78%.
  • Law-article prediction topped out at 25% for custody (LSTM/BiLSTM-TFIDF) and 50% for annulment (LR-word2vec).
  • Outcome-probability prediction:
    • custody: 88% with SVM and LR on word2vec;
    • annulment: about 68% with BiLSTM-TFIDF (the abstract says 69%).
  • Several LSTM/BiLSTM rows are identical across both feature types, with accuracy near 87.5% but F1 around 29%. This pattern suggests the models collapsed to the majority class.

Interesting ideas

  • I1. When real annotated cases are scarce, the authors added cases written by domain experts, a cheap way to cover rare outcome types. — For Mova (learn): Expert- or LLM-written controlled texts could fill gaps in silver data for rare constructions, provided they are tagged as synthetic and evaluated separately.
  • I2. The same case is modeled under two framings: one-sided input (pleading only) versus two-sided input (claim plus opposing answer). This shows how much the counter-argument changes the predicted outcome. — For Mova (frames): Frames could keep each party's assertions as separate speech acts, so a reader can reason over conflicting claims rather than over merged text.
  • I3. The verdict and its justification (law article or evidence) are predicted as separate targets, and the much lower score on justification exposes how weakly grounded the verdict predictions are. — For Mova (answer): Students should be scored on both the answer and its supporting trace, so a correct answer with a wrong justification counts as a partial failure.
  • I4. The tables show high accuracy alongside very low F1 for models that learned only the dominant class. This is an argument for always reporting per-class metrics next to accuracy. — For Mova (learn): Include a majority-class baseline as a negative control in every evaluation, and flag any model whose macro-F1 does not beat it.
  • I5. On a small dataset, simple linear models with sparse or averaged features outperformed recurrent networks. This is a reminder to establish strong cheap baselines before scaling up. — For Mova (tree): Keep the perceptron parser as a mandatory baseline when training gparse on small or new-dialect treebanks, and adopt the transformer only where it clearly wins.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Law

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.00238 · arXiv 2309.00238

@misc{abbara2023aljp,
  title = {{ALJP: An Arabic Legal Judgment Prediction in Personal Status Cases Using Machine Learning Models}},
  author = {Abbara, Salwa and Hafez, Mona and Kazzaz, Aya and Alhothali, Areej and Alsolami, Alhanouf},
  year = {2023},
  doi = {10.48550/arxiv.2309.00238},
  eprint = {2309.00238},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.00238}
}

References

25 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aletras, N., Tsarapatsanis, D., Preoţiuc-Pietro, D., Lampos, V.: Predicting judicial decisions of the european court of human rights: A natural language processing perspective. PeerJ Computer Science 2, 93 (2016) find
  2. Sil, R., Roy, A.: A novel approach on argument based legal prediction model using machine learning. In: 2020 International Conference on Smart Electronics and Communication (ICOSEC), pp. 487–490 (2020). IEEE find
  3. Medvedeva, M., Vols, M., Wieling, M.: Using machine learning to predict decisions of the european court of human rights. Artificial Intelligence and Law 28(2), 237–266 (2020) find
  4. Shaikh, R.A., Sahu, T.P., Anand, V.: Predicting outcomes of legal cases based on legal factors using classifiers. Procedia Computer Science 167, 2393–2402 (2020) find
  5. Zhang, H., Wang, X., Tan, H., Li, R.: Applying data discretization to dpcnn for law article prediction. In: NLPCC (2019) find
  6. Xiao, C., Zhong, H., Guo, Z., Tu, C., Liu, Z., Sun, M., Feng, Y., Han, X., Hu, Z., Wang, H., et al.: Cail2018: A large-scale legal dataset for judgment prediction. arXiv preprint arXiv:1807.02478 (2018) ↗ our card
  7. Li, S., Liu, B., Ye, L., Zhang, H., Fang, B.: Element-aware legal judgment prediction for criminal cases with confusing charges. In: 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), pp. 660–667 (2019). IEEE find
  8. Long, S., Tu, C., Liu, Z., Sun, M.: Automatic judgment prediction via legal reading comprehension. In: China National Conference on Chinese Computational Linguistics, pp. 558–572 (2019). Springer find
  9. Zhong, H., Guo, Z., Tu, C., Xiao, C., Liu, Z., Sun, M.: Legal judgment prediction via topological learning. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3540–3549 (2018) find our card
  10. Wang, C., Jin, X.: Study on the multi-task model for legal judgment prediction. In: 2020 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA), pp. 309–313 (2020). IEEE find
  11. Huang, D., Lin, W.: A model for legal judgment prediction based on multi-model fusion. In: 2019 3rd International Conference on Electronic Information Technology and Computer Engineering (EITCE), pp. 892–895 (2019). IEEE find
  12. Li, S., Zhang, H., Ye, L., Guo, X., Fang, B.: Mann: A multichannel attentive neural network for legal judgment prediction. IEEE Access 7, 151144–151155 (2019) find
  13. Yao, F., Sun, X., Yu, H., Yang, Y., Zhang, W., Fu, K.: Gated hierarchical multi-task learning network for judicial decision prediction. Neurocomputing 411, 313– 326 (2020) find
  14. Bao, Q., Zan, H., Gong, P., Chen, J., Xiao, Y.: Charge prediction with legal attention. In: CCF International Conference on Natural Language Processing and Chinese Computing, pp. 447–458 (2019). Springer find
  15. Sukanya, G., J.Priyadarshini: A meta analysis of attention models on legal judgment prediction system. International Journal of Advanced Computer Science and Applications 12(2) (2021) https://doi.org/10.14569/IJACSA.2021.0120266 ↗
  16. Yang, Z., Wang, P., Zhang, L., Shou, L., Xu, W.: A recurrent attention network for judgment prediction. In: International Conference on Artificial Neural Networks, pp. 253–266 (2019). Springer find
  17. Xu, N., Wang, P., Chen, L., Pan, L., Wang, X., Zhao, J.: Distinguish confusing law articles for legal judgment prediction. arXiv preprint arXiv:2004.02557 (2020) ↗ our card
  18. Kowsrihawat, K., Vateekul, P., Boonkwan, P.: Predicting judicial decisions of criminal cases from thai supreme court using bi-directional gru with attention mechanism. In: 2018 5th Asian Conference on Defense Technology (ACDT), pp. 50–55 (2018). IEEE find
  19. Yuan, L., Wang, J., Fan, S., Bian, Y., Yang, B., Wang, Y., Wang, X.: Automatic legal judgment prediction via large amounts of criminal cases. In: 2019 IEEE 5th International Conference on Computer and Communications (ICCC), pp. 2087–2091 (2019). IEEE find
  20. Wang, Y., Gao, J., Chen, J.: Deep learning algorithm for judicial judgment prediction based on bert. In: 2020 5th International Conference on Computing, Communication and Security (ICCCS), pp. 1–6 (2020). IEEE find
  21. Chalkidis, I., Androutsopoulos, I., Aletras, N.: Neural legal judgment prediction in english. arXiv preprint arXiv:1906.02059 (2019) ↗ our card
  22. Zhu, K., Guo, R., Hu, W., Li, Z., Li, Y.: Legal judgment prediction based on multiclass information fusion. Complexity 2020 (2020) find
  23. Guo, X., Zhang, H., Ye, L., Li, S., Zhang, G.: Tenrr: An approach based on innovative tensor decomposition and optimized ridge regression for judgment prediction of legal cases. IEEE Access 8, 167914–167929 (2020) find
  24. Guo, X., Zhang, H., Ye, L., Li, S.: Rnrtd: Intelligent approach based on the relationship-driven neural network and restricted tensor decomposition for multiple accusation judgment in legal cases. Computational intelligence and neuroscience 2019 (2019) find
  25. Soliman, A.B., Eissa, K., El-Beltagy, S.R.: Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science 117, 256–265 (2017) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abbara-2023-aljp-arabic-legal-judgment-prediction/