Elamur Research⌕
☰
home / papers / 2023 / DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services

DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services

Paper · 2023 · Shengbin Yue, Wei Chen, Siyuan Wang et al.

Summary

DISC-LawLLM targets a gap in Chinese legal AI. Earlier legal LLMs mostly handled plain question answering, but real legal work needs structured legal reasoning and accurate use of statutes that change over time. The authors built DISC-Law-SFT, a supervised fine-tuning collection of about 403K instructions. They drew on public Chinese legal NLP datasets, raw legal text such as court judgments and regulations, and existing open instruction sets. ChatGPT and human annotators rewrote these samples so that answers follow the legal syllogism: statute as major premise, case facts as minor premise, then a conclusion. A second data subset adds reference passages to teach the model to use retrieved evidence, and a retrieval module supplies statutes at inference time. The base model is Baichuan-13B. For evaluation they introduce DISC-Law-Eval, which combines graded multiple-choice exam questions with open questions scored by GPT-3.5. DISC-LawLLM beats prior legal LLMs and, on most exam subjects, GPT-3.5-Turbo. This suggests that carefully shaped domain data can matter more than model scale.

Key points

  • DISC-Law-SFT dataset has two parts:
    • DISC-Law-SFT-Pair (<input, output>) covers tasks such as element extraction, event detection, case classification, judgment prediction, similar-case matching, summarization, QA, reading comprehension and judicial exams.
    • DISC-Law-SFT-Triplet (<input, output, reference>) covers judgment prediction (16K) and legal QA (23K).
    • General data (Alpaca-GPT4 48K, Firefly 60K) is mixed in, for a total of roughly 403K samples.
  • Sources: task datasets include LEVEN, JointExtraction, CJRC, JEC-QA, CAIL2019, CAIL2020-sfzy and CAIL2022-yqzy. Raw text comes from China Judgements Online, laws and regulations, and online legal-advice platforms. Instruction data comes from Lawyer-LLaMa, LawGPT-zh and COIG-PC.
  • Model: Baichuan-13B fine-tuned with SFT. The authors say the recipe works for any decoder-only base model.
  • DISC-Law-Eval benchmark:
    • The objective part uses single- and multiple-answer exam questions split into Easy, Normal and Hard levels.
    • The subjective part scores Accuracy, Completeness and Clarity, using GPT-3.5 as the referee.
  • Results:
    • DISC-LawLLM averages about 7% higher accuracy than GPT-3.5-Turbo on most objective subjects.
    • On multi-answer questions for NJE and PAE, the gain over GPT-3.5-Turbo exceeds 50%.
    • In subjective evaluation, its average score is about 6% above ChatLaw.

Interesting ideas

  • I1. Syllogism-shaped targets. The authors rewrote training answers into a fixed reasoning template of rule, facts and conclusion. Any field with codified rules, such as customs regulations or compliance checks, could adopt the same pattern to make outputs easier to audit.
  • I2. Reference-carrying triplets. Some training examples include the supporting passage alongside the question and answer. This explicitly teaches the model to ground answers in supplied evidence, which is a cheap way to prepare a model for RAG rather than adding retrieval only at inference.
  • I3. Recasting annotated task datasets as instructions. Existing labelled corpora for extraction, classification or summarization were converted into instruction form with LLM and human help. This route is practical for low-resource languages that already have scattered task datasets.
  • I4. Mixing in general instruction data. General sets (Alpaca-GPT4, Firefly) sit alongside the legal data. This is a simple design choice that likely helps keep broad instruction-following while specializing the model.
  • I5. Grounded LLM-as-judge. The GPT-3.5 referee receives the reference answer, and scores are averaged over repeated runs. Both steps reduce judge bias and variance, and the approach transfers to any open-ended evaluation where human scoring is costly. The benchmark's difficulty tiers and single- versus multi-answer split also help show where reasoning, not just recall, breaks down.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations🎓Huazhong University of Science and TechnologyWuhan · China59 works · 8,310 citations🎓New York University ShanghaiShanghai · China11 works · 644 citations

Also on the byline: Northwest University of Politics and Law.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksPolitical Science and International Relations · 206 worksLaw · 84 works

🧭 Topics 1Research subjects and languages the work is about.Legal AI · 150 works

Cite

Cite as Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang and Zhongyu Wei. 2023. DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services. arXiv:2309.11325.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.11325 · arXiv 2309.11325

@misc{yue2023disc,
  title = {{DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services}},
  author = {Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang and Zhongyu Wei},
  year = {2023},
  doi = {10.48550/arxiv.2309.11325},
  eprint = {2309.11325},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.11325}
}

References

39 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baichuan-inc. 2023. Baichuan-13b. https://github.com/baichuan-inc/Baichuan-13B. ↗
  2. MJ Bommarito, Daniel Martin Katz, and E Detterman. 2018. Lexnlp: Natural language processing and information extraction for legal and regulatory texts. Research Handbook on Big Data Law. ↗
  3. CAIL. 2020. Cail2020. https://github.com/china-ai-law-challenge/CAIL2020. ↗
  4. CAIL. 2022. Cail2022. https://github.com/china-ai-law-challenge/CAIL2022. ↗
  5. Yanguang Chen, Yuanyuan Sun, Zhihao Yang, and Hongfei Lin. 2020. Joint entity and relation extraction for legal documents with legal feature enhancement. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1561– 1571, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  6. Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023a. Chatlaw: Open-source legal large language model with integrated external knowledge bases. find
  7. Yiming Cui, Ziqing Yang, and Xin Yao. 2023b. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177. ↗
  8. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335. ↗
  9. Xingyi Duan, Baoxin Wang, Ziyue Wang, Wentao Ma, Yiming Cui, Dayong Wu, Shijin Wang, Ting Liu, Tianxiang Huo, Zhen Hu, Heng Wang, and Zhiyuan Liu. 2019. Cjrc: A reliable human-annotated benchmark dataset for chinese judicial reading comprehension. In Chinese Computational Linguistics, pages 439–451, Cham. Springer International Publishing. find
  10. Anne von der Lieth Gardner. 1987. An artificial intelligence approach to legal reasoning. MIT press. find
  11. Quzhe Huang, Mingxu Tao, Zhenwei An, Chen Zhang, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023. Lawyer llama technical report. ArXiv, abs/2305.15062. ↗
  12. IDEA-CCNL. 2021. Fengshenbang-lm. https://github.com/IDEA-CCNL/Fengshenbang-LM. ↗
  13. Lu Ji, Zhongyu Wei, Xiangkun Hu, Yang Liu, Qi Zhang, and Xuan-Jing Huang. 2018. Incorporating argument-level interactions for persuasion comments evaluation using co-attention model. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3703–3714. find
  14. Lu Ji, Zhongyu Wei, Jing Li, Qi Zhang, and Xuanjing Huang. 2019. Discrete argument representation learning for interactive argument pair identification. arXiv preprint arXiv:1911.01621. ↗
  15. Cong Jiang and Xiaolei Yang. 2023. Legal syllogism prompting: Teaching large language models for legal judgment prediction. arXiv preprint arXiv:2307.08321. ↗
  16. Phi Manh Kien, Ha-Thanh Nguyen, Ngo Xuan Bach, Vu Tran, Minh Le Nguyen, and Tu Minh Phuong. 2020. Answering legal questions by learning neural attentive text representation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 988–998. find
  17. Haitao Li. 2023. Lexilaw. https://github.com/CSHaitao/LexiLaw. ↗
  18. Hongcheng Liu, Yusheng Liao, Yutong Meng, and Yuhao Wang. 2023. Lawgpt. https://github.com/LiuHC0428/LAW_GPT. ↗
  19. Yixiao Ma, Yunqiu Shao, Yueyue Wu, Yiqun Liu, Ruizhe Zhang, Min Zhang, and Shaoping Ma. 2021. Lecard: a legal case retrieval dataset for chinese law system. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 2342–2348. find
  20. Meta. 2023. Llama. https://github.com/facebookresearch/llama. ↗
  21. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. ↗
  22. OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. find
  23. OpenAI. 2023. Gpt-4 technical report.
  24. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277. ↗
  25. Richard A Posner. 1990. The problems of jurisprudence. Harvard University Press. find
  26. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506. find
  27. Pengxiao Song. 2023. Lawgpt. https://github.com/pengxiao-song/LaWGPT. ↗
  28. Yun Song and Zhongyu Wei. 2021. Inferring association between alcohol addiction and defendant’s emotion based on sound at court. Frontiers in Psychology, 12:669780. ↗
  29. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. ↗
  30. Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, and Jianfeng Xu. 2018. CAIL2018: A large-scale legal dataset for judgment prediction. CoRR, abs/1807.02478. ↗
  31. Jianxin Yang. 2023. Firefly. https://github.com/yangjianxin1/Firefly. ↗
  32. Wenmian Yang, Weijia Jia, Xiaojie Zhou, and Yutao Luo. 2019. Legal judgment prediction via multi-perspective bi-feedback network. arXiv preprint arXiv:1905.03969. ↗
  33. Feng Yao, Chaojun Xiao, Xiaozhi Wang, Zhiyuan Liu, Lei Hou, Cunchao Tu, Juanzi Li, Yun Liu, Weixing Shen, and Maosong Sun. 2022. LEVEN: A large-scale Chinese legal event detection dataset. In Findings of the Association for Computational Linguistics: ACL 2022, pages 183–201, Dublin, Ireland. Association for Computational Linguistics. find
  34. Hai Ye, Xin Jiang, Zhunchen Luo, and Wenhan Chao. 2018. Interpretable charge predictions for criminal cases: Learning to generate court views from fact descriptions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1854–1864, New Orleans, Louisiana. Association for Computational Linguistics. ymcui. 2023. Chinese-llama-alpaca-2. https://github.com/ymcui/Chinese-LLaMA-Alpaca-2. ↗
  35. Jian Yuan, Zhongyu Wei, Yixu Gao, Wei Chen, Yun Song, Donghua Zhao, Jinglei Ma, Zhen Hu, Shaokun Zou, Donghai Li, et al. 2021. Overview of smp-cail2020-argmine: The interactive argument-pair extraction in judgement document challenge. Data Intelligence, 3(2):287–307. ↗
  36. Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong, Yu Shu, Zhaoqun Li, Zekun Wang, Chenghua Lin, Wenhao Huang, and Jie Fu. 2023. Chinese open instruction generalist: A preliminary release. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. ↗
  37. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.
  38. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020a. How does nlp benefit legal system: A summary of legal artificial intelligence. arXiv preprint arXiv:2004.12158. ↗
  39. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020b. Jec-qa: A legal-domain question answering dataset. In Proceedings of AAAI. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yue-2023-disc-lawllm-fine-tuning-large/