Elamur Research⌕
☰
home / papers / 2026 / Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs' Legal Reasoning Capabilities

Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs' Legal Reasoning Capabilities

Paper · 2026 · Hongseok Oh, Wonseok Hwang and Kyoung-Woon On

Summary

Legal tasks mix domain knowledge with reasoning, so it is hard to tell whether an LLM fails because it lacks legal facts or because it reasons poorly. The authors address this with the Korean Canonical Legal Benchmark (KCL), built from Korean Bar Exam questions. Each question comes with supporting precedents drawn from expert solutions. Models are tested in two settings. A vanilla setting shows how much law they have memorized, and a setting with the precedents supplied isolates evidence-grounded reasoning.

KCL has two parts. KCL-MCQA contains multiple-choice items. KCL-Essay contains open-ended case questions, scored automatically by an LLM judge against rubrics written for each question.

Across more than 30 models, reasoning-oriented models consistently beat general-purpose ones. When precedents are provided, many frontier reasoning models exceed 80% on the multiple-choice part. The essay part stays difficult, with a best score of only 74.9%. The benchmark also targets Korean, a low-resource language, which helps move legal LLM evaluation beyond English.

Key points

  • KCL-MCQA: 283 five-option questions from the 2024–2025 Korean Bar Exams, with 1,103 supporting precedents (about 3.9 per question). The precedents were retrieved via the LBOX legal search engine.
  • KCL-Essay: 169 case-based questions from 2021–2025, with 550 precedents and 2,739 instance-level rubrics. The total is 2,905 points, weighted by the original exam scores.
  • Rubric creation: Rubrics were generated by Gemini 2.5 Flash from attorney-drafted seed examples. Three licensed attorneys reviewed 744 of them and had to fix only 13 (about 2%).
  • Judge validation: Gemini 2.5 Flash served as the judge. Its scores correlated with human experts at roughly 0.8 overall (Pearson) and about 0.9 for lower-point questions.
  • Model comparisons: Korean-specialized models such as A.X 4.0, Exaone 4.0 32B and HCX SEED Think 14B do well in the vanilla setting but gain less from the supplied precedents. Open-API models lead open-weight ones on reasoning-heavy tasks. Gemini 2.5 Pro scored best on the essay part.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SeoulSeoul · South Korea4 works · 284 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 worksLegal AI · 134 works

Cite

Cite as Hongseok Oh, Wonseok Hwang and Kyoung-Woon On. 2026. Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs' Legal Reasoning Capabilities. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-short.17

@inproceedings{oh2026korean,
  title = {{Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs' Legal Reasoning Capabilities}},
  author = {Hongseok Oh, Wonseok Hwang and Kyoung-Woon On},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-short.17},
  url = {https://doi.org/10.18653/v1/2026.eacl-short.17}
}

References

51 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 others. 2024. Phi-4 technical report. Preprint, arXiv:2412.08905. ↗
  2. Anthropic. 2025b. System card addendum: Claude opus 4.1. Accessed: 2025-10-02. find
  3. Anthropic. 2025c. System card: Claude opus 4 and claude sonnet 4. Accessed: 2025-10-02. find
  4. Anthropic. 2025d. System card: Claude sonnet 4.5. Accessed: 2025-10-02. find
  5. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330, Dublin, Ireland. Association for Computational Linguistics. ↗
  6. Odysseas S. Chlapanis, Dimitrios Galanis, and Ion Androutsopoulos. 2024. LAR-ECHR: A new legal argument reasoning task and dataset for cases of the European court of human rights. In Proceedings of the Natural Legal Language Processing Workshop 2024, pages 267–279, Miami, FL, USA. Association for Computational Linguistics. find
  7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168. ↗
  8. NAVER Corporation and NAVER Cloud Corporation. 2025. Hyperclova x seed 14b think model license agreement. find
  9. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, and 26 others. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261. ↗
  10. Google DeepMind. 2025a. Gemini 2.5 flash and 2.5 flash image model card. Accessed: 2025-10-02. find
  11. Google DeepMind. 2025b. Gemini 2.5 pro model card. Accessed: 2025-10-02. find
  12. DeepSeek. 2023. License-model. Accessed: 2025-10-02. find
  13. Moonshot AI. 2025. Kimi-k2 instruct license (modified mit). Accessed: 2025-10-02. find
  14. Anthropic. 2023. Anthropic on bedrock - commercial terms of service. Accessed: 2025-10-02. find
  15. Anthropic. 2025a. Claude 3.7 sonnet and claude code. Accessed: 2025-10-02. find
  16. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. ↗
  17. Yu Fan, Jingwei Ni, Jakob Merane, Etienne Salimbeni, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, and Joel Niklaus. 2025. Lexam: Benchmarking legal reasoning on 340 law exams. Preprint, arXiv:2505.12864. ↗
  18. Zhiyuan Fan, Weinong Wang, Xing W, and Debing Zhang. 2024. SedarEval: Automated evaluation using self-adaptive rubrics. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16916–16930, Miami, Florida, USA. Association for Computational Linguistics. find
  19. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. 2024. LawBench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7933–7962, Miami, Florida, USA. Association for Computational Linguistics. ↗
  20. Google. 2025a. Gemma terms of use. Last updated: September 12, 2025; Accessed: 2025-10-02. find
  21. Google. 2025b. Google apis terms of service. Accessed: 2025-10-02. find
  22. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Re, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, and 21 others. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  23. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z F Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081):633–638. find
  24. Wonseok Hwang, Dongjun Lee, Kyoungyeon Cho, Hanuhl Lee, and Minjoon Seo. 2022. A multi-task benchmark for korean legal language understanding and judgement prediction. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  25. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations. ↗
  26. Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, and 13 others. 2025. The BiGGen bench: A principled benchmark for fine-grained evaluation of language models with language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5877–5919, Albuquerque, New Mexico. Association for Computational Linguistics. find
  27. Yeeun Kim, Youngrok Choi, Eunkyung Choi, JinHwan Choi, Hai Jin Park, and Wonseok Hwang. 2024. Developing a pragmatic benchmark for assessing Korean legal language understanding in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5573–5595, Miami, Florida, USA. Association for Computational Linguistics.
  28. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611–626, New York, NY, USA. Association for Computing Machinery.
  29. Jihyung Lee, Daehui Kim, Seonjeong Hwang, Hyounghun Kim, and Gary Lee. 2025. KoBLEX: Open legal question answering with multi-hop reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4019–4053, Suzhou, China. Association for Computational Linguistics. find
  30. Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yiqun LIU. 2024. Lexeval: A comprehensive chinese legal benchmark for evaluating large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  31. Kaijing Ma, Xeron Du, Yunran Wang, Haoran Zhang, ZhoufutuWen, Xingwei Qu, Jian Yang, Jiaheng Liu, minghao liu, Xiang Yue, Wenhao Huang, and Ge Zhang. 2025. KOR-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks. In The Thirteenth International Conference on Learning Representations. find
  32. Meta. 2025a. Llama 4 community license agreement. Accessed: 2025-10-02. find
  33. AI Meta. 2025b. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta. com/blog/llama-4-multimodal-intelligence/, checked on, 4(7):2025. ↗
  34. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. gpt-oss-120b and gpt-oss-20b model card. Preprint, arXiv:2508.10925. ↗
  35. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card. Preprint, arXiv:2410.21276. ↗
  36. Kanana LLM Team, Yunju Bak, Hojin Lee, Minho Ryu, Jiyeon Ham, Seungjae Jung, Daniel Wontae Nam, Taegyeong Eo, Donghun Lee, Doohae Jung, Boseop Kim, Nayeon Kim, Jaesun Park, Hyunho Kim, Hyunwoong Ko, Changmin Lee, Kyoung-Woon On, Seulye Baeg, Junrae Cho, and 10 others. 2025b. Kanana: Compute-efficient bilingual language models. Preprint, arXiv:2502.18934. ↗
  37. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, and 150 others. 2025c. Kimi k2: Open agentic intelligence. Preprint, arXiv:2507.20534. ↗
  38. OpenAI. 2024a. Gpt-5 system card. Accessed: 2025-10-02. find
  39. OpenAI. 2024b. Openai terms of use. Accessed: 2025-10-02. find
  40. OpenAI. 2025a. Introducing GPT-4.1 in the api. Accessed: 2025-10-02. find
  41. OpenAI. 2025b. Openai o3 and o4-mini system card. System card, version published April 16, 2025. find
  42. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling. ↗
  43. LG AI Research, :, Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Yemuk Choi, Kyubeen Han, Seokhee Hong, Junwon Hwang, Taewan Hwang, Joonwon Jang, Hyojin Jeon, Kijeong Jeon, Gerrard Jeongwon Jo, Hyunjik Jo, Jiyeon Jung, Euisoon Kim, Hyosang Kim, and 23 others. 2025. Exaone 4.0: Unified large language models integrating non-reasoning and reasoning modes. Preprint, arXiv:2507.11407. ↗
  44. NAVER Cloud HyperCLOVA X Team. 2025. Hyperclova x think technical report. Preprint, arXiv:2506.22403. ↗
  45. Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization without overfitting: analyzing the training dynamics of large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.
  46. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗
  47. Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson, and Daniel E. Ho. 2025. A reasoning-focused legal retrieval benchmark. In Proceedings of the 2025 Symposium on Computer Science and Law, CSLAW ’25, page 169–193, New York, NY, USA. Association for Computing Machinery.
  48. Albert Sadowski and Jarosław A. Chudziak. 2025. On verifiable legal reasoning: A multi-agent framework with formalized knowledge representations. Preprint, arXiv:2509.00710. ↗
  49. Hayeon Seo. 2025. 슈퍼로이어, 국내 변호사시험 정 답률 74%로 합격점 돌파. Access Date: 2025-09-27. find
  50. SKT-AI. 2025. A.x-4.0 readme (english). Accessed: 2025-10-02. find
  51. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025a. Gemma 3 technical report. Preprint, arXiv:2503.19786. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oh-2026-korean-canonical-legal-benchmark-knowledge/