Elamur Research⌕
☰
home / papers / 2021 / Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents

Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents

Paper · 2021 · Chaojun Xiao, Xueyu Hu, Zhiyuan Liu et al.

Summary

Chinese court documents are long. Criminal and civil case facts average about 1,260 tokens, and nearly two thirds exceed 512 tokens, which is the limit of standard PLMs such as BERT and RoBERTa. Earlier legal PLMs kept full self-attention and so had to truncate these texts. The authors built Lawformer, a Longformer-based model that combines sliding-window, dilated sliding-window and global attention, so its cost grows linearly with input length. They continued MLM pre-training from RoBERTa-wwm-ext on tens of millions of criminal and civil judgments from China Judgment Online. Because existing judgment-prediction benchmarks covered only criminal cases and did not reflect real-world data, they also created a new dataset, CAIL-Long, with both criminal and civil cases. Lawformer was compared with BERT, RoBERTa and a legally pre-trained RoBERTa (L-RoBERTa) across four task types. It gave the clearest gains on long-input tasks such as judgment prediction and roughly matched L-RoBERTa on short-input tasks. The release offers a public long-context legal encoder and shows where long context actually pays off.

Key points

  • Pre-training corpus: 5,428,717 criminal documents (average 962.84 tokens, 17 GB) and 17,387,874 civil documents (average 1,353.03 tokens, 67 GB). Only documents whose fact description exceeds 50 tokens were kept.
  • Training setup: MLM objective, initialized from the RoBERTa-wwm-ext checkpoint, learning rate 5×10⁻⁵. Code and weights are available at github.com/thunlp/LegalPLMs.
  • CAIL-Long results: On criminal charge prediction, Lawformer reached Macro-F1 72.1 versus 70.8 for L-RoBERTa. It also had the lowest term-of-penalty distance (1.264 vs 1.280). On civil cause-of-action prediction it reached Macro 50.0 versus 49.4.
  • CJRC (reading comprehension): 9,532 questions over documents averaging only 441 tokens. Lawformer's joint F1 was 56.62 against 56.40 for L-RoBERTa, essentially a tie, since long documents were filtered out when the dataset was built.
  • JEC-QA (28,641 bar-exam multiple-choice questions): All models scored low, below about 28% accuracy on the full question set. The authors conclude that legal knowledge and reasoning remain open problems.

Interesting ideas

  • I1. Measure document length first. The authors plotted the token-length distribution of their target texts before choosing an architecture. This quick check shows whether a 512-token model is truly limiting for a new domain, for example Ukrainian court rulings or trade contracts.
  • I2. Choose global-attention tokens per task. Only [CLS] gets global attention for classification, while the whole question gets it for reading comprehension. This lets one pre-trained long encoder adapt cheaply to different downstream formats.
  • I3. Adapt an existing checkpoint instead of starting from scratch. Continuing pre-training from a short-context model (RoBERTa-wwm-ext) reuses its language knowledge and lowers compute cost. This suits languages or domains with limited resources.
  • I4. Benchmark on both short and long inputs. Including short-input tasks shows where the long-context architecture actually helps and where it merely ties a strong domain baseline. This keeps conclusions about model gains honest.
  • I5. Drop weak retrieved context. In JEC-QA, the authors left out BM25/TF-IDF reading materials because poor retrieval hurt accuracy. Noisy added context can do more harm than having none, so it is worth testing both setups.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓Beihang UniversityBeijing · China86 works · 9,097 citations🔬Beijing National Research Center for Information Science and TechnologyBeijing · China33 works · 6,706 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksPolitical Science and International Relations · 206 worksLaw · 84 works

🧭 Topics 1Research subjects and languages the work is about.Legal AI · 150 works

Cite

Cite as Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu and Maosong Sun. 2021. Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents. arXiv:2105.03887.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2105.03887 · arXiv 2105.03887

@misc{xiao2021lawformer,
  title = {{Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents}},
  author = {Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu and Maosong Sun},
  year = {2021},
  doi = {10.48550/arxiv.2105.03887},
  eprint = {2105.03887},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2105.03887}
}

References

40 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical bert embeddings. In Proceedings of Clinical Natural Language Processing Workshop, pages 72–78. ↗
  2. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of EMNLP-IJCNLP, pages 3606–3611. ↗
  3. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. ↗
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc-Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of NeurIPS. ↗
  5. Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural legal judgment prediction in English. In Proceedings of ACL, pages 4317–4323.
  6. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. Legal-bert:“preparing the muppets for court’”. In Proceedings of EMNLP: Findings, pages 2898– 2904. find
  7. Yen-Liang Chen, Yi-Hung Liu, and Wu-Liang Ho. 2013. A text mining approach to assist the general public in the retrieval of legal documents. Journal of the American Society for Information Science and Technology, 64(2):280–290. ↗
  8. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-training with whole word masking for Chinese bert. arXiv preprint arXiv:1906.08101. ↗
  9. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of ACL, pages 2978–2988. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171– 4186.
  11. Xingyi Duan, Baoxin Wang, Ziyue Wang, Wentao Ma, Yiming Cui, Dayong Wu, Shijin Wang, Ting Liu, Tianxiang Huo, Zhen Hu, et al. 2019a. Cjrc: A reliable human-annotated benchmark dataset for Chinese judicial reading comprehension. In Proceedings of CCL, pages 439–451. Springer. find
  12. Xinyu Duan, Yating Zhang, Lin Yuan, Xin Zhou, Xiaozhong Liu, Tianyi Wang, Ruocheng Wang, Qiong Zhang, Changlong Sun, and Fei Wu. 2019b. Legal summarization for multi-role debate dialogue via controversy focus mining and multi-task learning. In Proceedings of CIKM, pages 1361–1370. find
  13. Emad Elwany, Dave Moore, and Gaurav Oberoi. 2019. Bert goes to law school: Quantifying the competitive advantage of access to large legal corpora in contract understanding. In Proceedings of NeurIPS Workshop on Document Intelligence. ↗
  14. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of ACL, pages 8342–8360. find
  15. Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268. ↗
  16. Phi Manh Kien, Ha-Thanh Nguyen, Ngo Xuan Bach, Vu Tran, Minh Le Nguyen, and Tu Minh Phuong. 2020. Answering legal questions by learning neural attentive text representation. In Proceedings of COLING, pages 988–998. find
  17. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR. ↗
  18. Fred Kort. 1957. Predicting supreme court decisions mathematically: A quantitative analysis of the” right to counsel” cases. The American Political Science Review, 51(1):1–12. find
  19. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinform., 36(4):1234– 1240. ↗
  20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  21. Bingfeng Luo, Yansong Feng, Jianbo Xu, Xiang Zhang, and Dongyan Zhao. 2017. Learning to predict charges for criminal cases with legal basis. In Proceedings of EMNLP, pages 2727–2736. ↗
  22. Stuart S Nagel. 1963. Applying correlation analysis to case prediction. Tex. L. Rev., 42:1006. find
  23. Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. In Proceedings of ISCA Speech Synthesis Workshop, pages 125–125. ↗
  24. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237.
  25. Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, and Jie Tang. 2020. Blockwise self-attention for long document understanding. In Proceedings of EMNLP: Findings, pages 2555–2565. find
  26. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67. ↗
  27. K Raghav, P Krishna Reddy, and V Balakista Reddy. 2016. Analyzing the extraction of relevant legal judgments using paragraph-level and citation information. AI4JCArtificial Intelligence for Justice, page 30. find
  28. Jeffrey A Segal. 1984. Predicting supreme court cases probabilistically: The search and seizure cases, 1962-1981. The American political science review, pages 891–900. ↗
  29. Shohreh Shaghaghian, Luna Yue Feng, Borna Jafarpour, and Nicolai Pogrebnyakov. 2020. Customizing contextualized language models for legal document reviews. In Proceedings of IEEE Big Data, pages 2139–2148. IEEE. find
  30. Yunqiu Shao, Jiaxin Mao, Yiqun Liu, Weizhi Ma, Ken Satoh, Min Zhang, and Shaoping Ma. 2020. Bert-pli: Modeling paragraph-level interactions for legal case retrieval. In Proceedings of IJCAI, pages 3501– 3507. find
  31. Sainbayar Sukhbaatar, Édouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. In Proceedings of ACL, pages 331–335. ↗
  32. Yiquan Wu, Kun Kuang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Jun Xiao, Yueting Zhuang, Luo Si, and Fei Wu. 2020. De-biased court’s view generation with causality. In Proceedings of EMNLP, pages 763–780. find
  33. Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, et al. 2018. Cail2018: A large-scale legal dataset for judgment prediction. arXiv preprint arXiv:1807.02478. ↗
  34. Wenmian Yang, Weijia Jia, XIaojie Zhou, and Yutao Luo. 2019. Legal judgment prediction via multi-perspective bi-feedback network. In Proceedings of IJCAI. find
  35. Hai Ye, Xin Jiang, Zhunchen Luo, and Wenhan Chao. 2018. Interpretable charge predictions for criminal cases: Learning to generate court views from fact descriptions. In Proceedings of NAACL-HLT, pages 1854–1864. ↗
  36. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. In Proceedings of NeurIPS. ↗
  37. Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2018. Legal judgment prediction via topological learning. In Proceedings of EMNLP, pages 3540–3549.
  38. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020a. How does NLP benefit legal system: A summary of legal artificial intelligence. In Proceedings of ACL, pages 5218–5230.
  39. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020b. Jec-qa: A legal-domain question answering dataset. In Proceedings of the AAAI, volume 34, pages 9701– 9708. find
  40. Haoxi Zhong, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2019. Open Chinese language pre-trained model zoo. Technical report. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2021-lawformer-pre-trained-language-model/