Elamur Research
☰
home / papers / 2019 / Neural Legal Judgment Prediction in English

Neural Legal Judgment Prediction in English

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1424.pdf

Summary

The paper brings neural legal judgment prediction to English. This task means forecasting a court's decision from a description of case facts. Earlier English work used only feature-based linear models on about 600 cases. The authors release a much larger public ECHR dataset of roughly 11.5k European Court of Human Rights cases. It includes raw fact texts, the violated Convention articles and the court's own importance scores. They define three tasks: binary violation detection, multi-label prediction of violated articles, and regression of case importance. They compare BiGRU-Att, HAN, LWAN and BERT against BOW-SVM/SVR and trivial baselines. Because BERT accepts only 512 wordpieces, they introduce HIER-BERT, which encodes each fact separately and then aggregates the facts with attention. Neural models beat the bag-of-words approach in all tasks, and the hierarchical models (HAN, HIER-BERT) perform best. The authors also anonymize the data to check whether predictions depend on demographic details. The work matters because it supplies an open benchmark for this task in English. It also shows how to apply pretrained transformers to long legal documents.

Key points

  • ECHR dataset: 7,100 train / 1,380 dev / 2,998 test cases, covering 66 Convention articles and protocols. Train and dev (1959–2013) are balanced between violation and non-violation. The test set (2014–2018) has about 66% violations. Of the 66 labels, 45 never appear in training, which makes the dataset useful for few-shot experiments.
  • Binary violation (macro-F1):
    • HIER-BERT scores 82.0, HAN 80.5 and BiGRU-Att 79.5, compared with 71.8 for BOW-SVM.
    • Plain BERT drops to 17.0 because the case texts must be truncated.
    • After anonymization, the scores fall only slightly (HAN 80.2, HIER-BERT 80.1).
  • Multi-label violation (micro-F1): HIER-BERT (60.0) and HAN (59.9) lead, ahead of LWAN (57.6) and BOW-SVM (50.4). All models do poorly on rare labels with fewer than 50 training examples.
  • Case importance (scale 1–4): the MAJORITY baseline gets the lowest MAE (.369) because the scores are heavily skewed. However, it cannot separate important cases from unimportant ones. HIER-BERT reaches the best Spearman's ρ (.527), a moderate correlation. This suggests that the facts alone are not enough to predict importance.
  • Limitations: attention maps show partial interpretability, including possible bias toward place names such as Kharkiv, but they do not amount to trustworthy legal justifications.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Law

Cited by

229 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 222 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1424

@inproceedings{chalkidis2019neural,
  title = {{Neural Legal Judgment Prediction in English}},
  author = {Chalkidis, Ilias and Androutsopoulos, Ion and Aletras, Nikolaos},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1424},
  url = {https://doi.org/10.18653/v1/p19-1424}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nikolaos Aletras et al. 2016. Predicting judicial decisions of the European Court of Human Rights: a Natural Language Processing perspective. PeerJ Computer Science, 2:e93. find
  2. Reuben Binns, Max Van Kleek, Michael Veale, Ulrik Lyngs, Jun Zhao, and Nigel Shadbolt. 2018. ‘it’s reducing a human being to a percentage’: Perceptions of justice in algorithmic decisions. In CHI, pages 377:1–377:14. find
  3. Cristian Cardellino, Milagro Teruel, Laura Alonso Alemany, and Serena Villata. 2017. Legal NERC with ontologies, Wikipedia and curriculum learning. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 254–259. find
  4. Ilias Chalkidis, Ion Androutsopoulos, and Achilleas Michos. 2017. Extracting contract elements. In Proceedings of the 16th International Conference on Artificial Intelligence and Law (ICAIL), pages 19–28, London, UK. find
  5. Ilias Chalkidis, Ion Androutsopoulos, and Achilleas Michos. 2018. Obligation and prohibition extraction using hierarchical RNNs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 254–259. find
  6. Ilias Chalkidis, Emmanouil Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2019. Extreme multi-label legal text classification: A case study in EU legislation. In Proceedings of the Natural Legal Language Processing Workshop (NLLP) of NAACL-HLT, pages 78–87. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT). find
  8. Finale Doshi-Velez and Ben Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv. find our card
  9. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AIS-TATS), pages 249–256. find
  10. Daniel Martin Katz. 2012. Quantitative legal prediction-or-how I learned to stop worrying and start preparing for the data-driven future of the legal services industry. Emory Law Journal, 62:909. find
  11. Diederik P. Kingma and Jim Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR). find
  12. Reed C Lawlor. 1963. What computers can do: Analysis and prediction of judicial decisions. American Bar Association Journal, pages 337–344. find
  13. Bingfeng Luo, Yansong Feng, Jianbo Xu, Xiang Zhang, and Dongyan Zhao. 2017. Learning to predict charges for criminal cases with legal basis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (Conference on Empirical Methods in Natural Language Processing (EMNLP)), pages 2727–2736. find
  14. Masha Medvedeva, Michel Vols, and Martijn Wieling. 2018. Judicial decisions of the European Court of Human Rights: Looking into the crystal ball. In Proceedings of the Conference on Empirical Legal Studies. find
  15. James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018. Explainable prediction of medical codes from clinical text. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 1101–1111. find our card
  16. Ramesh Nallapati and Christopher D. Manning. 2008. Legal docket classification: Where machine learning stumbles. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (Conference on Empirical Methods in Natural Language Processing (EMNLP)), pages 438–446. find
  17. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  18. Dru Stevenson and Nicholas J Wagoner. 2015. Bargaining in the shadow of big data. Florida Law Review, 67:1337. find
  19. Zikun Hu, Xiang Li, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2018. Few-shot charge prediction with discriminative legal attributes. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 487–498. find
  20. Octavia-Maria Şulea, Marcos Zampieri, Mihaela Vela, and Josef van Genabith. 2017. Predicting the law area and decisions of French Supreme Court cases. In Proceedings of the Recent Advances in Natural Language Processing (RANLP), pages 716–722. find
  21. Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT). find our card
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS). find
  23. William Yang Wang, Elijah Mayfield, Suresh Naidu, and Jeremiah Dittmar. 2012. Historical analysis of legal opinions with a sparse mixed-effects latent variable model. In ACL, pages 740–749. find
  24. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning (ICML), pages 2048–2057. find our card
  25. Zichao Yang et al. 2016. Hierarchical attention networks for document classification. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 1480–1489. find our card
  26. Hai Ye, Xin Jiang, Zhunchen Luo, and Wenhan Chao. 2018. Interpretable charge predictions for criminal cases: Learning to generate court views from fact descriptions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (Conference on Empirical Methods in Natural Language Processing (EMNLP)), pages 1854–1864. find
  27. Haoxi Zhong, Guo Zhipeng, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2018. Legal judgment prediction via topological learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3540–3549. find our card