Elamur Research⌕
☰
home / papers / 2025 / LAiW: A Chinese Legal Large Language Models Benchmark

LAiW: A Chinese Legal Large Language Models Benchmark

Paper · 2025 · Yongfu Dai, Duanyu Feng, Jimin Huang et al.

Summary

Chinese legal LLMs are usually judged task by task, for example on question answering or consultation. Such scores ignore whether a model actually follows the legal syllogism, the reasoning chain lawyers expect: legal norm (major premise), case facts (minor premise) and judgment (conclusion). This gap makes legal professionals reluctant to trust these systems. The authors propose LAiW, a benchmark that arranges 14 tasks, rebuilt from public Chinese LegalAI datasets, into three increasingly demanding levels. The levels cover retrieving basic information (FIR), inferring the applicable legal principles (LPI) and advanced legal application (ALA). They evaluate general-purpose and law-specific models automatically, then ask legal experts to compare generated answers. Strong models write fluent, well-formed answers to complex applied tasks. Yet they perform poorly on the basic retrieval and inference steps, so their apparent reasoning does not reflect real syllogistic logic. Expert judgments broadly agree with the automatic scores. The results suggest that legal-domain pre-training, not instruction tuning alone, is the more promising route. This matters for anyone building domain benchmarks that experts in a field need to trust.

Key points

  • Three-level taxonomy tied to the legal syllogism:
    • FIR covers tasks such as legal article recommendation, element recognition, NER, judicial summarization and case recognition.
    • LPI covers tasks such as controversy focus mining, similar case matching and charge prediction.
    • ALA covers tasks such as judicial reasoning generation and legal consultation.
  • Data sources: tasks are adapted from CAIL2018–CAIL-2021, CJRC and LAIC-2021, among others. The comparison table lists roughly 11k instances, versus DISC-Law-Eval (7 tasks, 3k) and LawBench (20 tasks, 10k).
  • Models evaluated (18 in total):
    • 11 general models, including GPT-4, ChatGPT, Baichuan2-Chat, ChatGLM, Llama/Llama2-Chat, Chinese-LLaMA and Ziya-LLaMA.
    • 7 legal models: HanFei, wisdomInterrogatory, Fuzi-Mingcha, LexiLaw, LaWGPT, Lawyer-LLaMA and ChatLaw.
  • Expert evaluation: three legal experts rated answers using win rates.
    • In Judicial Reasoning Generation, Fuzi-Mingcha led (automatic score 63.58, win rate 0.65), followed by HanFei (60.13, 0.59).
    • GPT-4 and ChatGPT led in Legal Consultation.
  • Effect of fine-tuning vs. pre-training: models fine-tuned on applied tasks (e.g., Lawyer-LLaMA) generate well-normalized text but can lose ground on FIR and LPI compared with their base models. In-context learning did not fix this. Models with heavy legal pre-training (HanFei, Fuzi-Mingcha) held up best on syllogistic ability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Wuhan UniversityWuhan · China47 works · 1,246 citations🎓Sichuan UniversityChengdu · China20 works · 1,293 citations🎓Southwest Petroleum UniversityChengdu · China2 works · 35 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksPolitical Science and International Relations · 206 worksLaw · 84 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,685 worksEvaluation & benchmarks · 1,341 worksLegal AI · 150 works

Cite

Cite as Yongfu Dai, Duanyu Feng, Jimin Huang, Haochen Jia, Qianqian Xie, Yifang Zhang, Weiguang Han, Wei Tian and Hao Wang. 2025. LAiW: A Chinese Legal Large Language Models Benchmark. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{dai2025laiw,
  title = {{LAiW: A Chinese Legal Large Language Models Benchmark}},
  author = {Yongfu Dai, Duanyu Feng, Jimin Huang, Haochen Jia, Qianqian Xie, Yifang Zhang, Weiguang Han, Wei Tian and Hao Wang},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.716/}
}

References

58 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baichuan. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305. ↗
  2. Sheng Bi, Zafar Ali, Tianxing Wu, and Guilin Qi. 2024. Knowledge-enhanced model with dual-graph interaction for confusing legal charge prediction. Expert Systems with Applications, 249:123626. ↗
  3. Due to the significant amount of work required to construct this benchmark and complete the evaluation, we also acknowledge the following three Yu Cao, Yuanyuan Sun, Ce Xu, Chunnan Li, Jinming limitations: Du, and Hongfei Lin. 2022. Cailie 1.0: A dataset for 1) In the manual evaluation experiment, to save challenge of ai in law-information extraction v1. 0. the workload, only a portion of the data and the AI Open, 3:208–212. 10746 find
  4. Ilias Chalkidis, Nicolas Garneau, Catalina Goanta, Daniel Katz, and Anders Søgaard. 2023. LeXFiles and LegalLAMA: Facilitating English multinational legal language model development. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15513–15535, Toronto, Canada. Association for Computational Linguistics.
  5. Davide Chicco and Giuseppe Jurman. 2020. The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC genomics, 21:1–13. ↗
  6. Jonathan H Choi. 2023. How to use large language models for empirical legal research. Journal of Institutional and Theoretical Economics (Forthcoming). ↗
  7. Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023a. Chatlaw: Open-source legal large language model with integrated external knowledge bases. find
  8. Junyun Cui, Xiaoyu Shen, and Shaochun Wen. 2023b. A survey on legal judgment prediction: Datasets, metrics, models and challenges. IEEE Access. ↗
  9. Yiming Cui, Ziqing Yang, and Xin Yao. 2023c. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177. ↗
  10. Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. 2024. Large legal fictions: Profiling legal hallucinations in large language models. arXiv preprint arXiv:2401.01301. ↗
  11. Wentao Deng, Jiahuan Pei, Keyi Kong, Zhe Chen, Furu Wei, Yujun Li, Zhaochun Ren, Zhumin Chen, and Pengjie Ren. 2023. Syllogistic reasoning for legal judgment analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13997–14009, Singapore. Association for Computational Linguistics. find
  12. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335. ↗
  13. Xingyi Duan, Baoxin Wang, Ziyue Wang, Wentao Ma, Yiming Cui, Dayong Wu, Shijin Wang, Ting Liu, Tianxiang Huo, Zhen Hu, et al. 2019. Cjrc: A reliable human-annotated benchmark dataset for chinese judicial reading comprehension. In Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18, pages 439–451. Springer. find
  14. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback.
  15. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. Lawbench: Benchmarking legal knowledge of large language models. arXiv preprint arXiv:2309.16289. ↗
  16. Duanyu Feng, Bing Hu, Yifang Zhang, Wei Tian, and Hao Wang. 2023. Multi-scale heterogeneous graph attention network for prison term prediction. In 2023 IEEE 3rd International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA), volume 3, pages 1395–1404. IEEE. find
  17. Yi Feng, Chuanyi Li, and Vincent Ng. 2022. Legal judgment prediction: A survey of the state of the art. IJCAI. ijcai. org, pages 5461–9. find
  18. Jidong Ge, Yunyun Huang, Xiaoyu Shen, Chuanyi Li, and Wei Hu. 2021. Learning fine-grained fact-article correspondence in legal cases. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3694–3706. ↗
  19. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. ↗
  20. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. ↗
  21. Zikun Hu, Xiang Li, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2018. Few-shot charge prediction with discriminative legal attributes. In Proceedings of the 27th International Conference on Computational Linguistics, pages 487–498.
  22. Quzhe Huang, Mingxu Tao, Zhenwei An, Chen Zhang, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023a. Lawyer llama technical report. ArXiv, abs/2305.15062. ↗
  23. Yue Huang, Lijuan Sun, Chong Han, and Jian Guo. 2023b. A high-precision two-stage legal judgment summarization. Mathematics, 11(6):1320. ↗
  24. Ambedkar Kanapala, Sukomal Pal, and Rajendra Pamula. 2019. Text summarization from legal documents: a survey. Artificial Intelligence Review, 51:371–402. find
  25. Hyungjin Ko and Jaewook Lee. 2023. Can chatgpt improve investment decision? from a portfolio management perspective. From a Portfolio Management Perspective. ↗
  26. Aditya Kuppa, Nikon Rasumov-Rahe, and Marc Voses. 2023. Chain of reference prompting helps llm to think like a lawyer. In Generative AI+ Law Workshop. find
  27. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. ↗
  28. Hongcheng Liu, Yusheng Liao, Yutong Meng, and Yuhao Wang. 2023. Lawgpt: 中文法律对话语言模 型. https://github.com/LiuHC0428/LAW_GPT. ↗
  29. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. ↗
  30. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  31. Arianna Trozze, Toby Davies, and Bennett Kleinberg. 2023. Large language models in cryptocurrency securities cases: Can chatgpt replace lawyers? arXiv preprint arXiv:2308.06032. ↗
  32. Daniel Locke and Guido Zuccon. 2022. Case law retrieval: problems, methods, challenges and evaluations in the last 20 years. arXiv preprint arXiv:2202.07209. ↗
  33. Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. 2023. Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975. ↗
  34. Luyao Ma, Yating Zhang, Tianyi Wang, Xiaozhong Liu, Wei Ye, Changlong Sun, and Shikun Zhang. 2021. Legal judgment prediction with multi-stage case representation learning in the real court setting. In Proceedings of the 44th International ACM SI-GIR Conference on Research and Development in Information Retrieval, pages 993–1002. find
  35. Zhu Wang. 2022. 司法人工智能推理辅助的“准三段 论”实现路径. 政法论坛, 40(05):28–39. find
  36. Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. 2023. Lextreme: A multi-lingual and multi-task benchmark for the legal domain. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics. ↗
  37. Jibao Wen and Wanwei He. 2023. Hanfei. https://github.com/siat-nlp/HanFei. ↗
  38. Jerzy Wróblewski. 1974. Legal syllogism and rationality of judicial decision. Rechtstheorie, 5:33. find
  39. Shiguang Wu, Zhongkun Liu, Zhen Zhang, Zheng Chen, Wentao Deng, Wenhao Zhang, Jiyuan Yang, Zhitao Yao, Yougang Lyu, Xin Xin, Shen Gao, Pengjie Ren, Zhaochun Ren, and Zhumin Chen. 2023. fuzi.mingcha. https://github.com/irlab-sdu/ fuzi.mingcha. ↗
  40. OpenAI. 2023. Gpt-4 technical report.
  41. Edwin W Patterson. 2013. Logic in the law. In Logic, Probability, and Presumptions in Legal Reasoning, pages 287–321. Routledge. find
  42. Song Pengxiao, Zhou Zhi, and cainiao. 2023. Lawgpt: 基 于 中 文 法 律 知 识 的 大 语 言 模 型. https://github.com/pengxiao-song/LaWGPT. ↗
  43. Yiquan Wu, Kun Kuang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Jun Xiao, Yueting Zhuang, Luo Si, and Fei Wu. 2020. De-biased court’s view generation with causality. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 763–780. find
  44. Carlo Sansone and Giancarlo Sperlí. 2022. Legal information retrieval systems: State-of-the-art and open issues. Information Systems, 106:101967. ↗
  45. Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, et al. 2018. Cail2018: A large-scale legal dataset for judgment prediction. arXiv preprint arXiv:1807.02478. ↗
  46. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615. ↗
  47. Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Tianyang Zhang, Xianpei Han, Zhen Hu, Heng Wang, et al. 2019. Cail2019-scm: A dataset of similar case matching in legal domain. arXiv preprint arXiv:1911.08962. ↗
  48. Quinten Steenhuis, David Colarusso, and Bryce Willey. 2023. Weaving pathways for justice with gpt: Llm-driven automated drafting of interactive legal applications. arXiv preprint arXiv:2312.09198. ↗
  49. Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. Pixiu: A large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443. ↗
  50. Linan Yue, Qi Liu, Binbin Jin, Han Wu, and Yanqing An. 2024. A circumstance-aware neural framework for explainable legal judgment prediction. IEEE Transactions on Knowledge and Data Engineering. ↗
  51. Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, and Zhongyu Wei. 2023. Disc-lawllm: Fine-tuning large language models for intelligent legal services. ↗
  52. Dian Zhang, Hewei Zhang, Long Wang, Jiamei Cui, Wen Zheng, et al. 2022a. Recognition of chinese legal elements based on transfer learning and semantic relevance. ↗
  53. Wireless Communications and Mobile Computing, 2022. find
  54. Jiaxing Zhang, Ruyi Gan, Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaoqun Dong, Junqing He, Jianheng Zhuo, Qi Yang, Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang, Hao Wang, Xiaojun Wu, Zhongshen Zeng, and Chongpei Chen. 2022b. Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence. CoRR, abs/2209.02970. ↗
  55. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223. ↗
  56. Haoxi Zhong, Yuzhong Wang, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020a. Iteratively questioning and answering for interpretable legal judgment prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1250–1257. find
  57. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020b. How does nlp benefit legal system: A summary of legal artificial intelligence. arXiv preprint arXiv:2004.12158. ↗
  58. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020c. Jec-qa: a legal-domain question answering dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9701–9708. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dai-2025-laiw-chinese-legal-large-language/