Elamur Research
☰
home / papers / 2018 / CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction

CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1807.02478

Summary

Legal Judgment Prediction (LJP) asks a system to read the facts of a case and infer the court's decision. Earlier work was limited by small, often private datasets restricted to narrow case types. To address this, the authors built CAIL2018, a public Chinese corpus of criminal cases drawn from China Judgments Online, the repository of the Supreme People's Court. They kept only judgment documents with a single defendant and removed rare charges and law articles. Each remaining case pairs a fact description with three targets: the applicable law articles, the charge, and the prison term. Labels were extracted with regular expressions. The authors tested three standard text classifiers: TFIDF+SVM, FastText, and CNN. Overall accuracy is high for charges and articles. However, macro-averaged precision and recall are much lower, and prison-term prediction is clearly harder. This exposes the effect of label imbalance. The resource matters because it gives LJP research a large, openly released benchmark several times bigger than earlier datasets.

Key points

  • CAIL2018 contains 2,676,075 criminal cases labeled with 183 law articles, 202 charges, and prison terms, filtered from 5,730,302 raw documents.
  • Labels are heavily skewed: the 10 most common charges account for 79.0% of cases, while the 10 rarest cover only 0.12%.
  • The data was split into 1,710,856 training and 965,219 test cases. Text was segmented with THULAC, and neural models used 200-dimensional Skip-Gram embeddings.
  • CNN achieved the best accuracy: 97.6% on charges, 97.6% on law articles, and 78.2% on prison terms. Macro-precision and macro-recall stayed far lower, often below 50%.
  • The dataset and baselines were released through the CAIL competition (cail.cipsc.org.cn).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Law

Cited by

190 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 183 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1807.02478 · arXiv 1807.02478

@misc{xiao2018cail2018,
  title = {{CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction}},
  author = {Xiao, Chaojun and Zhong, Haoxi and Guo, Zhipeng and Tu, Cunchao and Liu, Zhiyuan and Sun, Maosong and Feng, Yansong and Han, Xianpei and Hu, Zhen and Wang, Heng and Xu, Jianfeng},
  year = {2018},
  doi = {10.48550/arxiv.1807.02478},
  eprint = {1807.02478},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1807.02478}
}

References

21 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preotiuc-Pietro, and Vasileios Lampos. 2016. Predicting judicial decisions of the european court of human rights: A natural language processing perspective. PeerJ Computer Science 2. find
  2. Zikun Hu, Xiang Li, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2018. Few-shot charge prediction with discriminative legal attributes. In Proceedings of COLING. find
  3. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of EACL. find
  4. R Keown. 1980. Mathematical models for legal prediction. Computer/LJ 2:829. find
  5. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP. find our card
  6. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR. find
  7. Fred Kort. 1957. Predicting supreme court decisions mathematically: A quantitative analysis of the ”right to counsel” cases. American Political Science Review 51(1):1–12. find
  8. Benjamin E Lauderdale and Tom S Clark. 2012. The supreme court’s many median justices. American Political Science Review 106(4):847–866. Charge 故意伤害 intentional injury Prison Term 12个月 12 months Defendant 胡某 Miss./Mr. Hu Relevant Articles Acc.% MP% MR% 93.3 45.8 38.1 92.9 71.8 52.4 97.6 37.4 21.8 Terms of Penalty Acc.% MP% MR% 74.6 48.0 24.5 75.4 75.4 46.1 78.2 45.5 36.1 find
  9. Wan-Chen Lin, Tsung-Ting Kuo, Tung-Jia Chang, Chueh-An Yen, Chao-Ju Chen, and Shou-de Lin. 2012. Exploiting machine learning models for chinese legal documents labeling, case classification, and sentencing prediction. In Processdings of RO-CLING. page 140. find
  10. Chao-Lin Liu and Chwen-Dar Hsieh. 2006. Exploring phrase-based classification of judicial documents for criminal charges in chinese. In Proceedings of IS-MIS. pages 681–690. find
  11. Yi Hung Liu and Yen Liang Chen. 2017. A two-phase sentiment analysis approach for judgement prediction. Journal of Information Science . find
  12. Bingfeng Luo, Yansong Feng, Jianbo Xu, Xiang Zhang, and Dongyan Zhao. 2017. Learning to predict charges for criminal cases with legal basis. In Proceedings of EMNLP. find
  13. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS. pages 3111–3119. find
  14. Stuart S Nagel. 1963. Applying correlation analysis to case prediction. Tex. L. Rev. 42:1006. find
  15. Gerard Salton and Christopher Buckley. 1988. Term-weighting approaches in automatic text retrieval. Information processing & management 24(5):513– 523. find
  16. Jeffrey A Segal. 1984. Predicting supreme court cases probabilistically: The search and seizure cases, 1962-1981. American Political Science Review 78(4):891–900. find
  17. Octavia Maria Sulea, Marcos Zampieri, Mihaela Vela, and Josef Van Genabith. 2017. Exploring the use of text classi cation in the legal domain. In Proceedings of ASAIL workshop. find
  18. Maosong Sun, Xinxiong Chen, Kaixu Zhang, Zhipeng Guo, and Zhiyuan Liu. 2016. Thulac: An efficient lexical analyzer for chinese. . find
  19. Johan AK Suykens and Joos Vandewalle. 1999. Least squares support vector machine classifiers. Neural processing letters 9(3):293–300. find
  20. Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In Proceedings of EMNLP. pages 1422–1432. find
  21. S Sidney Ulmer. 1963. Quantitative analysis of judicial processes: Some practical and theoretical applications. Law & Contemp. Probs. 28:164. find