Elamur Research⌕
☰
home / papers / 2026 / OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demand

OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demand

Paper · 2026 · Sergio Servantez, Sarah B. Lawsky, Rajiv Jain et al.

Summary

Static legal question-answer benchmarks are costly to annotate. They also reduce model behaviour to one accuracy number, which makes it hard to tell which reasoning skill actually failed. OpenExempt tackles this with a generator built on U.S. Bankruptcy Code (Title 11) asset-exemption law. Legal experts encoded federal and state exemption statutes and a pool of debtor assets in paired forms: natural-language text and symbolic structure. From a configuration file, the framework assembles bankruptcy cases, writes them up as prompts, and computes gold answers with a deterministic branch-and-bound solver. No one has to annotate solutions by hand. Users can vary one difficulty factor at a time, such as asset count, domicile history, distractor statements or task scope. The authors used it to build a released benchmark and tested 13 language models. Performance often held up until a specific point and then fell sharply, mainly on long, globally constrained reasoning chains and when obfuscating statements were added. The approach shows how a configurable, symbolically grounded benchmark can locate where models break.

Key points

  • OpenExempt Benchmark: 9,765 samples in nine suites. Six are diagnostic suites that vary a single axis; three are competency suites at basic, intermediate and advanced levels. Both the framework and the data are public under CC BY 4.0 on GitHub and Hugging Face.
  • Knowledge base: more than 500 expert-labelled assets, together with the federal exemption rules and the exemption laws of five states (Wisconsin, Pennsylvania, Oregon, Illinois and Arizona). The task is framed as a knapsack-like optimisation that minimises the value of unprotected assets.
  • Domicile scaling: reasoning models hit their clearest breaking point at four domiciles, where every reasoning model lost at least 0.145 F1.
  • Asset scaling (2→8 assets): Llama-4-Scout lost 0.057 F1 on EC and 0.053 on EV, but 0.177 on NA and 0.348 on OE. GPT-5 lost only 0.034 on OE.
  • Gold intermediate steps can hurt: giving models correct intermediate answers sometimes lowered their scores. Four of six reasoning models did worse on NA when handed gold EC/EV results, an effect strongest in GPT-5, o3 and Gemini-2.5-Pro.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States184 works · 69,372 citations🎓Northwestern UniversityEvanston · United States41 works · 2,194 citations🏢Adobe SystemsSan Jose · United States35 works · 3,711 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 worksLegal AI · 134 works

Cite

Cite as Sergio Servantez, Sarah B. Lawsky, Rajiv Jain, Daniel W. Linna Jr. and Kristian J Hammond. 2026. OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demand. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1328

@inproceedings{servantez2026openexempt,
  title = {{OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demand}},
  author = {Sergio Servantez, Sarah B. Lawsky, Rajiv Jain, Daniel W. Linna Jr. and Kristian J Hammond},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1328},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1328}
}

References

46 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, M Saiful Bari, and Haidar Khan. 2024. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. Preprint, arXiv:2402.01781. ↗
  2. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Accessed: 2025-12-25. find
  3. Anthropic. 2025. Claude sonnet 4: System card. Accessed: 2025-12-25. find
  4. Andrew Blair-Stanek, Nils Holzenberger, and Benjamin Van Durme. 2023. Can gpt-3 perform statutory reasoning? Preprint, arXiv:2302.06100. ↗
  5. Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural legal judgment prediction in English. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317–4323, Florence, Italy. Association for Computational Linguistics.
  6. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. 2022. Lexglue: A benchmark dataset for legal language understanding in english. Preprint, arXiv:2110.00976. ↗
  7. Christopher D. Clack, Vikram A. Bakshi, and Lee Braine. 2017. Smart contract templates: foundations, design landscape and research directions. Preprint, arXiv:1608.00771. ↗
  8. Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2022. Introduction to Algorithms, 4 edition. MIT Press, Cambridge, MA. find
  9. Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. 2024. Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1):64–93. ↗
  10. DeepSeek. 2025. Deepseek r1 model card. Accessed: 2025-12-27. find
  11. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025a. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. ↗
  12. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025b. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. ↗
  13. Yu Fan, Jingwei Ni, Jakob Merane, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Etienne Salimbeni, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, and Joel Niklaus. 2025. Lexam: Benchmarking legal reasoning on 340 law exams. Preprint, arXiv:2505.12864. ↗
  14. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. Lawbench: Benchmarking legal knowledge of large language models. Preprint, arXiv:2309.16289. ↗
  15. Google. 2025a. Gemini 2.5 flash model card. Accessed: 2025-12-25. find
  16. Google. 2025b. Gemini 2.5 pro model card. Accessed: 2025-12-25. Neel Guha, Julian Nyarko, Daniel E. Ho, and Christopher Ré. Building genai benchmarks: A case study in legal applications. find
  17. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, and 21 others. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Preprint, arXiv:2308.11462. ↗
  18. Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. 2022. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. Preprint, arXiv:2207.00220. ↗
  19. Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. Cuad: An expert-annotated nlp dataset for legal contract review. Preprint, arXiv:2103.06268. ↗
  20. Valentin Hofmann, David Heineman, Ian Magnusson, Kyle Lo, Jesse Dodge, Maarten Sap, Pang Wei Koh, Chun Wang, Hannaneh Hajishirzi, and Noah A. Smith. 2025. Fluid language model benchmarking. Preprint, arXiv:2509.11106. ↗
  21. Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2020. A dataset for statutory reasoning in tax law entailment and question answering. Preprint, arXiv:2005.05257. ↗
  22. Liane Huttner and Denis Merigoux. 2020. Catala: Moving towards the future of legal expert systems. Artificial Intelligence and Law. find
  23. Abhinav Joshi, Akshat Sharma, Sai Kiran Tanikella, and Ashutosh Modi. 2023. U-creat: Unsupervised case retrieval using events extraction. Preprint, arXiv:2307.05260. ↗
  24. Sarah Lawsky. 2017. A logic for statutes. Florida Tax Review, 21:60–80. ↗
  25. Sarah Lawsky. 2022. Coding the code: Catala and computationally accessible tax law. SMU Law Review, 75:535. ↗
  26. Denis Merigoux, Nicolas Chataing, and Jonathan Protzenko. 2021. Catala: a programming language for the law. Proc. ACM Program. Lang., 5(ICFP). ↗
  27. Meta. 2025. Llama 4 model cards and prompt formats. Accessed: 2025-12-25. find
  28. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2025. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. Preprint, arXiv:2410.05229. ↗
  29. Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. 2023. Lextreme: A multi-lingual and multi-task benchmark for the legal domain. In Findings of the Association for Computational Linguistics: EMNLP 2023, page 3016–3054. Association for Computational Linguistics. ↗
  30. Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, and Daniel E. Ho. 2024. Multilegalpile: A 689gb multilingual legal corpus. Preprint, arXiv:2306.02069. ↗
  31. Joel Niklaus, Lucia Zheng, Arya D. McCarthy, Christopher Hahn, Brian M. Rosen, Peter Henderson, Daniel E. Ho, Garrett Honke, Percy Liang, and Christopher Manning. 2025. Lawinstruct: A resource for studying language model adaptation to the legal domain. Preprint, arXiv:2404.02127. ↗
  32. OpenAI. 2025a. Gpt-5 system card. Accessed: 2025-12-25. find
  33. OpenAI. 2025b. Introducing gpt-4.1 in the api. Accessed: 2025-12-25. find
  34. OpenAI. 2025c. Openai o3 and o4-mini system card. Accessed: 2025-12-25. find
  35. Pydantic. 2025. Pydantic validation. Accessed: 2025-12-28. find
  36. RapidFuzz. 2025. Rapidfuzz documentation. Accessed: 2025-12-28. find
  37. Niall Roche, Walter Hernandez, Eason Chen, Jérôme Siméon, and Dan Selman. 2021. Ergo–a programming language for smart legal contracts. arXiv preprint arXiv:2112.07064. ↗
  38. Sergio Servantez, Joe Barrow, Kristian Hammond, and Rajiv Jain. 2024. Chain of logic: Rule-based reasoning with large language models. Preprint, arXiv:2402.10400. ↗
  39. Sergio Servantez, Nedim Lipka, Alexa Siu, Milan Aggarwal, Balaji Krishnamurthy, Aparna Garimella, Kristian Hammond, and Rajiv Jain. 2023. Computable contracts by extracting obligation logic graphs. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL ’23, page 267–276, New York, NY, USA. Association for Computing Machinery. find
  40. Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. 2025. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. Preprint, arXiv:2506.06941. ↗
  41. Harry Surden. 2012. Computable contracts. UC Davis Law Review, 46. find
  42. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786. ↗
  43. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Preprint, arXiv:2406.01574. ↗
  44. Li Zhang, Matthias Grabmair, Morgan Gray, and Kevin Ashley. 2025. Thinking longer, not always smarter: Evaluating llm capabilities in hierarchical legal reasoning. Preprint, arXiv:2510.08710. ↗
  45. Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. 2021. When does pretraining help? assessing self-supervised learning for law and the casehold dataset. Preprint, arXiv:2104.08671. ↗
  46. Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson, and Daniel E. Ho. 2025. A reasoning-focused legal retrieval benchmark. In Proceedings of the Symposium on Computer Science and Law on ZZZ, CSLAW ’25, page 169–193. ACM.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/servantez-2026-openexempt-diagnostic-benchmark-legal-reasoning/