Elamur Research⌕
☰
home / papers / 2024 / LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models

LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models

Paper · 2024 · Haitao Li, You Chen, Qingyao Ai et al.

Summary

LLMs are increasingly used by lawyers and judges, yet legal work demands accuracy, reliability and fairness that these models do not reliably provide. Existing Chinese evaluations either target general or school-level knowledge (e.g., C-Eval) or repackage traditional NLP datasets (LawBench, LaiW). Such repackaging mixes inconsistent metrics and largely ignores legal ethics. The authors build LexEval, a Chinese legal benchmark structured around a new LexAbility Taxonomy, whose six Bloom-inspired levels start with Memorization and Understanding at the base, move through Logic Inference and Discrimination, and end with Generation as well as Ethic. The data combines three sources: reformatted existing datasets, real exam questions, and new items annotated by legal experts. All tasks use standardized metrics. The authors test 38 general and legal-specific LLMs in zero-shot and few-shot settings. Even GPT-4, the strongest model, falls well short of dependable legal assistance. Models are weakest at memorizing legal knowledge, discriminating legal content, generating legal texts and handling ethics. The benchmark and its leaderboard are public and meant to grow.

Key points

  • Scale: 23 tasks and 14,150 questions. The authors present it as the largest Chinese legal evaluation set. It was published in the NeurIPS 2024 Datasets and Benchmarks track.
  • Task examples by level:
    • Memorization: legal concepts, rules and how laws evolved.
    • Understanding: element recognition, fact verification, reading comprehension, relation extraction and NER.
    • Logic Inference: cause, article and penalty prediction, multi-hop reasoning, legal calculation and argument mining.
    • Discrimination: similar case identification and document proofreading.
    • Generation: summaries, judicial analysis, legal translation and open-ended QA.
    • Ethic: bias and discrimination, morality and privacy.
  • Metrics: accuracy for most levels and ROUGE-L for Generation tasks.
  • Few-shot results among the top 10 models: GPT-4 ranks first with a 53.7 average, followed by Qwen-14B-Chat at 49.9.
  • Few-shot gains are inconsistent. GPT-4 improves while Qwen-14B-Chat gets worse, which the authors attribute partly to overly long prompts.
  • Legal-specific LLMs are also weak at Memorization. This suggests that simply adding legal corpora to pre-training is not sufficient.

Interesting ideas

  • I1. Organize tasks by cognitive ability, not by dataset or difficulty. Grouping tasks into a hierarchy (recall, comprehension, reasoning, evaluation, generation, ethics) lets developers see which layer of competence is failing. This could structure any domain benchmark, for example a Ukrainian legal or administrative one.
  • I2. Blend three data sources. Mixing recycled datasets, authentic exam questions and freshly expert-annotated items balances cost, realism and coverage. Exams reflect how professionals are actually tested, and new annotations fill gaps that older datasets miss.
  • I3. Unify formats and metrics. Converting heterogeneous datasets into a common format with standard metrics makes models comparable across tasks. It also makes adding new tasks much easier.
  • I4. Treat ethics as a first-class evaluation level. Bias, morality and privacy are scored alongside knowledge tasks. This puts safety-relevant behavior in the same report as competence, which matters wherever model outputs affect people.
  • I5. Compare zero-shot and few-shot side by side. Reporting both settings, with per-model up and down markers, shows that in-context examples are an unreliable way to inject domain knowledge. This check is cheap and worth including in other evaluations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksPolitical Science and International Relations · 206 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksLegal AI · 150 works

Cite

Cite as Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang and Yiqun Liu. 2024. LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models. arXiv:2409.20288.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.20288 · arXiv 2409.20288

@misc{li2024lexeval,
  title = {{LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models}},
  author = {Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang and Yiqun Liu},
  year = {2024},
  doi = {10.48550/arxiv.2409.20288},
  eprint = {2409.20288},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.20288}
}

References

54 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. ↗
  2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. ↗
  3. BELLEGroup. Belle: Be everyone’s large language model engine. https://github.com/LianjiaTech/BELLE, 2023. ↗
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  5. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. Lexglue: A benchmark dataset for legal language understanding in english. arXiv preprint arXiv:2110.00976, 2021. ↗
  6. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. ↗
  7. Inyoung Cheong, King Xia, KJ Feng, Quan Ze Chen, and Amy X Zhang. (a) i am not a lawyer, but...: Engaging legal experts towards responsible llm policies for legal advice. arXiv preprint arXiv:2402.01864, 2024. ↗
  8. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. find
  9. Zhumin Chu, Qingyao Ai, Yiteng Tu, Haitao Li, and Yiqun Liu. Pre: A peer review based large language model evaluator, 2024. find
  10. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. ↗
  11. Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. Chatlaw: Open-source legal large language model with integrated external knowledge bases, 2023. find
  12. Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177, 2023. ↗
  13. Yongfu Dai, Duanyu Feng, Jimin Huang, Haochen Jia, Qianqian Xie, Yifang Zhang, Weiguang Han, Wei Tian, and Hao Wang. Laiw: A chinese legal large language models benchmark (a technical report). arXiv preprint arXiv:2310.05620, 2023. ↗
  14. Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. How ready are pre-trained abstractive models and llms for legal case judgement summarization? arXiv preprint arXiv:2306.01248, 2023. ↗
  15. Qian Dong, Yiding Liu, Qingyao Ai, Haitao Li, Shuaiqiang Wang, Yiqun Liu, Dawei Yin, and Shaoping Ma. I3 retriever: Incorporating implicit interaction in pre-trained language models for passage retrieval. CIKM ’23, page 441–451, New York, NY, USA, 2023. Association for Computing Machinery. find
  16. Qian Dong, Yiding Liu, Qingyao Ai, Zhijing Wu, Haitao Li, Yiqun Liu, Shuaiqiang Wang, Dawei Yin, and Shaoping Ma. Unsupervised large language model alignment for information retrieval via contrastive feedback. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 48–58, New York, NY, USA, 2024. Association for Computing Machinery. find
  17. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. Lawbench: Benchmarking legal knowledge of large language models. arXiv preprint arXiv:2309.16289, 2023. ↗
  18. Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681–694, 2020. find
  19. Neel Guha, Julian Nyarko, Daniel E Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N Rockmore, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. arXiv preprint arXiv:2308.11462, 2023. ↗
  20. Wanwei He, Jiabao Wen, Lei Zhang, Hao Cheng, Bowen Qin, Yunshui Li, Feng Jiang, Junying Chen, Benyou Wang, and Min Yang. Hanfei-1.0. https://github.com/siat-nlp/HanFei, 2023. ↗
  21. Quzhe Huang, Mingxu Tao, Zhenwei An, Chen Zhang, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. Lawyer llama technical report. arXiv preprint arXiv:2305.15062, 2023. ↗
  22. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023. ↗
  23. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. Gpt-4 passes the bar exam. Available at SSRN 4389233, 2023. find
  24. David R Krathwohl. A revision of bloom’s taxonomy: An overview. Theory into practice, 41(4):212–218, 2002. find
  25. Haitao Li, Qingyao Ai, Jia Chen, Qian Dong, Yueyue Wu, Yiqun Liu, Chong Chen, and Qi Tian. Sailer: Structure-aware pre-trained language model for legal case retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 1035–1044, New York, NY, USA, 2023. Association for Computing Machinery.
  26. Haitao Li, Qingyao Ai, Jia Chen, Qian Dong, Zhijing Wu, Yiqun Liu, Chong Chen, and Qi Tian. Blade: Enhancing black-box large language models with small domain-specific models, 2024. find
  27. Haitao Li, Qingyao Ai, Xinyan Han, Jia Chen, Qian Dong, Yiqun Liu, Chong Chen, and Qi Tian. Delta: Pre-train a discriminative encoder for legal case retrieval via structural word alignment, 2024. find
  28. Haitao Li, Qingyao Ai, Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Zheng Liu, and Zhao Cao. Constructing tree-based index for efficient and effective dense retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 131–140, New York, NY, USA, 2023. Association for Computing Machinery. find
  29. Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yujia Zhou, Qian Dong, and Yiqun Liu. Calibraeval: Calibrating prediction distribution to mitigate selection bias in llms-as-judges, 2024.
  30. Haitao Li, Yunqiu Shao, Yueyue Wu, Qingyao Ai, Yixiao Ma, and Yiqun Liu. Lecardv2: A large-scale chinese legal case retrieval dataset. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 2251–2260, New York, NY, USA, 2024. Association for Computing Machinery. find
  31. Haitao Li, Weihang Su, Changyue Wang, Yueyue Wu, Qingyao Ai, and Yiqun Liu. Thuir@coliee 2023: Incorporating structural knowledge into pre-trained language models for legal case retrieval, 2023. find
  32. Zihao Li. The dark side of chatgpt: Legal and ethical challenges from stochastic parrots and hallucination. arXiv preprint arXiv:2304.14347, 2023. ↗
  33. Yixiao Ma, Yunqiu Shao, Yueyue Wu, Yiqun Liu, Ruizhe Zhang, Min Zhang, and Shaoping Ma. Lecard: a legal case retrieval dataset for chinese law system. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 2342–2348, 2021. find
  34. Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models, 2023. find
  35. John J. Nay, David Karamardian, Sarah B. Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H. Choi, and Jungo Kasai. Large language models as tax attorneys: A case study in legal capabilities emergence, 2023.
  36. OpenAI. Gpt-4 technical report, 2023.
  37. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. ↗
  38. Jaromir Savelka, Kevin D. Ashley, Morgan A Gray, Hannes Westermann, and Huihui Xu. Can gpt-4 support analysis of textual data in tasks requiring highly specialized domain expertise?, 2023. find
  39. Yunqiu Shao, Haitao Li, Yueyue Wu, Yiqun Liu, Qingyao Ai, Jiaxin Mao, Yixiao Ma, and Shaoping Ma. An intent taxonomy of legal case retrieval. ACM Trans. Inf. Syst., 42(2), December 2023. find
  40. Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Hang Yan, Xiangyang Liu, Yunfan Shao, Qiong Tang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, Lingling Wu, Zhangyue Yin, Xuanjing Huang, and Xipeng Qiu. Moss: Training conversational language models from synthetic data. 2023. find
  41. Zhongxiang Sun. A short survey of viewing large language models in legal aspect. arXiv preprint arXiv:2303.09136, 2023. ↗
  42. InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM, 2023. ↗
  43. MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. Accessed: 2023-03-28. find
  44. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. ↗
  45. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. ↗
  46. Shiguang Wu, Zhongkun Liu, Zhen Zhang, Zheng Chen, Wentao Deng, Wenhao Zhang, Jiyuan Yang, Zhitao Yao, Yougang Lyu, Xin Xin, Shen Gao, Pengjie Ren, Zhaochun Ren, and Zhumin Chen. fuzi.mingcha. https://github.com/irlab-sdu/fuzi.mingcha, 2023. ↗
  47. Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, and Jianfeng Xu. Cail2018: A large-scale legal dataset for judgment prediction, 2018.
  48. Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiangsheng Li, Haitao Li, Yiqun Liu, and Jin Ma. T2ranking: A large-scale chinese benchmark for passage ranking. SIGIR ’23, page 2681–2690, New York, NY, USA, 2023. Association for Computing Machinery. find
  49. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023. ↗
  50. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022. ↗
  51. Jiaxing Zhang, Ruyi Gan, Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaoqun Dong, Junqing He, Jianheng Zhuo, Qi Yang, Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang, Hao Wang, Xiaojun Wu, Zhongshen Zeng, and Chongpei Chen. Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence. CoRR, abs/2209.02970, 2022. ↗
  52. Ruizhe Zhang, Haitao Li, Yueyue Wu, Qingyao Ai, Yiqun Liu, Min Zhang, and Shaoping Ma. Evaluation ethics of llms in legal domain, 2024. find
  53. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. Jec-qa: a legal-domain question answering dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9701–9708, 2020. find
  54. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2024-lexeval-comprehensive-chinese-legal-benchmark/