Elamur Research⌕
☰
home / papers / 2021 / ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation

ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation

Summary

India's courts carry a large backlog of pending cases. A tool that suggests likely outcomes could help judges, but only if it also gives reasons that legal professionals can follow. The authors build ILDC, a corpus of roughly 35k English-language proceedings of the Supreme Court of India, collected from indiankanoon.org. Each case is labelled with whether the claim was accepted or rejected. A separate test subset was also marked up by legal experts with the passages that justify each ruling. On this basis they define CJPE. A system receives the full unstructured case text, with the final judgment removed, and must predict the binary outcome and point to supporting text. No explanation labels are available for training. The authors benchmark a range of baseline classifiers and propose a hierarchical occlusion method for explanations. The best predictor reaches 78% accuracy, compared with 94% for human experts. Machine explanations overlap only modestly with expert rationales. Long, noisy legal documents therefore remain a hard, open problem.

Key points

  • ILDC contains 34,816 Supreme Court of India documents, covering all case types rather than only criminal or civil cases. Party and judge identities are anonymized.
  • The ILDC multi split has 32,305 training, 994 validation and 1,517 test documents. The ILDC expert set has 56 documents with gold explanations. Average lengths range from about 2.9k to 3.9k tokens.
  • The best prediction model reaches 78% accuracy, while legal experts reach 94%.
  • Explanations are compared with expert annotations using ROUGE-1/2/L, Jaccard, BLEU, METEOR, Overlap-Max and Overlap-Min. Overlap-Min agreement is high, but the best ROUGE-L score is only 0.4445 (with Expert 4).
  • The corpus and the code for prediction and explanation models are released on GitHub (Exploration-Lab/CJPE).

Interesting ideas

  • I1. Explanations without explanation supervision. The training data carries only outcome labels, and gold rationales exist only in a small expert-annotated test set. This keeps annotation costs low while still allowing faithful evaluation of interpretability. The same setup suits any domain where rationales are expensive to collect.
  • I2. Hierarchical occlusion for long inputs. The model masks pieces of a document at several levels of granularity to see which parts drive a prediction. This works for texts far longer than a transformer's input window. It could be reused for long reports, contracts or parliamentary transcripts.
  • I3. Human expert ceiling on the same task. Measuring expert accuracy (94%) next to model accuracy shows how much headroom remains and how hard the task really is. Without it, 78% would be hard to interpret.
  • I4. Several overlap metrics side by side. Lenient measures like Overlap-Min suggest strong agreement, while ROUGE-style measures reveal a large gap. Reporting both prevents an overly optimistic reading of explanation quality.
  • I5. Realistic, unstructured input. Prediction uses whole case documents instead of pre-extracted facts. This mirrors real use and exposes difficulties that cleaned benchmarks hide, a useful principle when designing new evaluation datasets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Law

Cited by

101 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 97 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.313

@inproceedings{malik2021ildc,
  title = {{ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation}},
  author = {Malik, Vijit and Sanjay, Rishabh and Nigam, Shubham Kumar and Ghosh, Kripabandhu and Guha, Shouvik Kumar and Bhattacharya, Arnab and Modi, Ashutosh},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.313},
  url = {https://doi.org/10.18653/v1/2021.acl-long.313}
}

References

57 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preotiuc-Pietro, and Vasileios Lampos. 2016. Predicting judicial decisions of the European Court of Human Rights:A Natural Language Processing perspective. PeerJ Computer Science, 2:93. find
  2. Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140. find
  3. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. ↗
  4. Paheli Bhattacharya, Kaustubh Hiware, Subham Rajgaria, Nilay Pochhi, Kripabandhu Ghosh, and Saptarshi Ghosh. 2019a. A comparative study of summarization algorithms applied to legal case judgments. In European Conference on Information Retrieval, pages 413–428. Springer. find
  5. Paheli Bhattacharya, Shounak Paul, Kripabandhu Ghosh, Saptarshi Ghosh, and Adam Wyner. 2019b. Identification of Rhetorical Roles of Sentences in Indian Legal Judgments. In Legal Knowledge and Information Systems - JURIX 2019, volume 322 of Frontiers in Artificial Intelligence and Applications, pages 3–12. IOS Press. find
  6. Richard Eckart de Castilho, Eva Mujdricza-Maydt, Seid Muhie Yimam, Silvana Hartmann, Iryna Gurevych, Anette Frank, and Chris Biemann. 2016. A web-based tool for the integrated annotation of semantic and syntactic structures. In Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH), pages 76–84. find
  7. Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural Legal Judgment Prediction in English. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317–4323, Florence, Italy. Association for Computational Linguistics. find our card
  8. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The Muppets straight out of Law School. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898– 2904, Online. Association for Computational Linguistics. ↗ our card
  9. Huajie Chen, Deng Cai, Wei Dai, Zehui Dai, and Yadong Ding. 2019. Charge-Based Prison Term Prediction with Deep Gating Network. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6362–6367, Hong Kong, China. Association for Computational Linguistics. find
  10. Pooja Chitkara, Ashutosh Modi, Pravalika Avvaru, Sepehr Janghorbani, and Mubbasir Kapadia. 2019. Topic Spotting using Hierarchical Networks with Self Attention. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3755–3761, Minneapolis, Minnesota. Association for Computational Linguistics. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  12. J. L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 75(5). find
  13. Filippo Galgani, Paul Compton, and Achim Hoffmann. 2012. Towards automatic generation of catchphrases for legal case reports. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 414–425. Springer. find
  14. Zikun Hu, Xiang Li, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2018. Few-Shot Charge Prediction with Discriminative Legal Attributes. In Proceedings of the 27th International Conference on Computational Linguistics, pages 487–498, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
  15. Peter Jackson, Khalid Al-Kofahi, Alex Tyrrell, and Arun Vachher. 2003. Information extraction from case law and retrieval of prior cases. Artificial Intelligence, 150(1-2):239–290. find
  16. Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  17. Xin Jiang, Hai Ye, Zhunchen Luo, WenHan Chao, and Wenjia Ma. 2018. Interpretable Rationale Augmented Charge Prediction System. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 146–151, Santa Fe, New Mexico. Association for Computational Linguistics. find
  18. Justice Markandey Katju. 2019. Backlog of cases crippling judiciary. https://tinyurl.com/ v4xu6mvk . ↗
  19. Daniel Martin Katz, Michael J. Bommarito, II, and Josh Blackman. 2017. A general approach for predicting the behavior of the Supreme Court of the United States. PLOS ONE, 12:1–18. find
  20. Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar. Association for Computational Linguistics. ↗ our card
  21. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The Efficient Transformer. In International Conference on Learning Representations. find our card
  22. Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Jonathan Reynolds, Alexander Melnikov, Natalia Lunova, and Orion Reblitz-Richardson. 2019. Pytorch Captum. https://github.com/pytorch/captum . ↗
  23. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the second workshop on statistical machine translation, pages 228–231. ↗ our card
  24. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International conference on machine learning, pages 1188– 1196. find our card
  25. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220. ↗
  26. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81. find
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  28. Shangbang Long, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2019. Automatic judgment prediction via legal reading comprehension. In China National Conference on Chinese Computational Linguistics, pages 558–572. Springer. find
  29. Edward Loper and Steven Bird. 2002. NLTK: the natural language toolkit. arXiv preprint cs/0205028. find our card
  30. Bingfeng Luo, Yansong Feng, Jianbo Xu, Xiang Zhang, and Dongyan Zhao. 2017. Learning to Predict Charges for Criminal Cases with Legal Basis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2727–2736, Copenhagen, Denmark. Association for Computational Linguistics. find
  31. Arpan Mandal, Kripabandhu Ghosh, Arindam Pal, and Saptarshi Ghosh. 2017. Automatic catchphrase identification from legal court case documents. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 2187– 2190. find
  32. Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018. Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram features. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 528–540, New Orleans, Louisiana. Association for Computational Linguistics. find our card
  33. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. ↗ our card
  34. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825–2830. find
  35. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  36. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. ”Why should I trust you?” Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144. find our card
  37. Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: High-Precision Model-Agnostic Explanations. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), pages 1527–1535. AAAI Press. find
  38. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. ↗
  39. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In International Conference on Machine Learning, pages 3145–3153. PMLR. find our card
  40. Benjamin Strickson and Beatriz de la Iglesia. 2020. Legal Judgement Prediction for UK Courts. ICISS 2020: The 3rd International Conference on Information Science and System, Cambridge, UK, March 19-22, 2020, pages 204–209. find
  41. Octavia-Maria Şulea, Marcos Zampieri, Mihaela Vela, and Josef van Genabith. 2017. Predicting the Law Area and Decisions of French Supreme Court Cases. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 716–722, Varna, Bulgaria. INCOMA Ltd. find
  42. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319–3328. PMLR. find
  43. Vu Tran, Minh Le Nguyen, and Ken Satoh. 2019. Building Legal Case Retrieval Systems with Lexical Matching and Summarization Using A Pre-Trained Phrase Scoring Model. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law, ICAIL ’19, page 275–282, New York, NY, USA. Association for Computing Machinery. find
  44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  45. Fact Descriptions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1854–1864, New Orleans, Louisiana. Association for Computational Linguistics. find
  46. Pengfei Wang, Yu Fan, Shuzi Niu, Ze Yang, Yongfeng Zhang, and Jiafeng Guo. 2019. Hierarchical Matching Network for Crime Classification. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 325–334. find
  47. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗ our card
  48. Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, et al. 2018. Cail2018: A large-scale legal dataset for judgment prediction. arXiv preprint arXiv:1807.02478. ↗ our card
  49. Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818–833. Springer. find our card
  50. Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2018. Legal Judgment Prediction via Topological Learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3540–3549, Brussels, Belgium. Association for Computational Linguistics. find our card
  51. Haoxi Zhong, Yuzhong Wang, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. Iteratively Questioning and Answering for Interpretable Legal Judgment Prediction. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):1250–1257. find
  52. Ning Xie, Gabrielle Ras, Marcel van Gerven, and Derek Doran. 2020. Explainable deep learning: A field guide for the uninitiated. arXiv preprint arXiv:2004.14545. ↗
  53. Nuo Xu, Pinghui Wang, Long Chen, Li Pan, Xiaoyan Wang, and Junzhou Zhao. 2020. Distinguish Confusing Law Articles for Legal Judgment Prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3086–3095, Online. Association for Computational Linguistics. find our card
  54. Wenmian Yang, Weijia Jia, Xiaojie Zhou, and Yutao Luo. 2019a. Legal Judgment Prediction via Multi-Perspective Bi-Feedback Network. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 4085– 4091. find
  55. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019b. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753– 5763. find our card
  56. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480–1489. find our card
  57. Hai Ye, Xin Jiang, Zhunchen Luo, and Wenhan Chao. 2018. Interpretable Charge Predictions for Criminal Cases: Learning to Generate Court Views from find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/malik-2021-ildc-cjpe-indian-legal-documents/