Elamur Research⌕
☰
home / papers / 2025 / A Reasoning-Focused Legal Retrieval Benchmark

A Reasoning-Focused Legal Retrieval Benchmark

Paper · 2025 · Lucia Zheng, Neel Guha, Javokhir Arifov et al.

Summary

Legal AI tools increasingly pair LLMs with retrievers over case law and statutes, but realistic English benchmarks for testing such pipelines are scarce. Existing legal retrieval sets usually come from citation links between court opinions. As a result, query and target text often share a lot of vocabulary, queries do not look like questions lawyers actually ask, and answers are rarely provided for end-to-end evaluation. The authors address this with two new resources:

  • Bar Exam QA: hypothetical bar-exam fact patterns linked to supporting passages from judicial opinions.
  • Housing Statute QA: practical housing-law questions linked to statutes from many jurisdictions.

Together they contain about 10K query–gold passage–answer triples. Law students and researchers annotated the passages by hand, following a workflow that mimics real legal research. Because queries and relevant documents have little word overlap, lexical retrievers such as BM25 perform poorly. A query-expansion method that first generates a legal reasoning rollout improves retrieval. Downstream QA gains are limited, because models struggle to use even ideal passages. The work suggests legal retrievers need genuine reasoning ability.

Key points

  • Downstream format: answers are multiple-choice, so factual correctness of retrieval-augmented LLMs can be scored automatically. This contrasts with open-ended generation tasks such as CLERC.
  • Corpus size: Housing Statute QA searches roughly 1–2M documents, far more than the ~10K–100K used in reasoning retrieval benchmarks like BRIGHT and BIRCO.
  • Retrievers tested: BM25, E5-small/base/large-v2 and E5-mistral-7b-instruct. Recall@10 drops as query–passage lexical similarity decreases.
  • Upper bound for Llama 3 8B Instruct: supplying the gold passage raised accuracy by only about 20%. Retrieval improvements therefore translate into small QA gains.
  • GPT-4o-mini results:
    • On Housing Statute QA, retrieved passages added 23.53 points over using no passage.
    • Retrieved passages reached 71.71% accuracy, versus 68.51% when the reasoning rollout served as a pseudo-passage.
    • On Bar Exam QA, the rollout used as a pseudo-passage lowered accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations🎓Princeton UniversityPrinceton · United States90 works · 39,109 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksPolitical Science and International Relations · 182 works

🧭 Topics 4Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 worksInformation retrieval & search · 473 worksLegal AI · 134 works

Cite

Cite as Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson and Daniel E. Ho. 2025. A Reasoning-Focused Legal Retrieval Benchmark. arXiv:2505.03970.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3709025.3712219 · arXiv 2505.03970

@misc{zheng2025reasoning,
  title = {{A Reasoning-Focused Legal Retrieval Benchmark}},
  author = {Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson and Daniel E. Ho},
  year = {2025},
  doi = {10.1145/3709025.3712219},
  eprint = {2505.03970},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3709025.3712219}
}

References

43 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021). ↗
  2. Ilias Chalkidis, Manos Fergadiotis, Nikolaos Manginas, Eva Katakalou, and Prodromos Malakasiotis. 2021. Regulatory Compliance through Doc2Doc Information Retrieval: A case study in EU/UK legislation where text similarity has limitations. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, Online, 3498–3511. https://doi.org/10.18653/v1/2021.eacl-main.305 ↗
  3. Andrea Anne Curcio, Carol L Chomsky, and Eileen R Kaufman. 2018. How to Build a Better Bar Exam. New York State Bar Association Journal (2018), 37–41. find
  4. Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. 2024. Large legal fictions: Profiling legal hallucinations in large language models. arXiv preprint arXiv:2401.01301 (2024). ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https://doi.org/10.18653/v1/N19-1423 ↗
  6. Randy Goebel, Yoshinobu Kano, Mi-Young Kim, Juliano Rabelo, Ken Satoh, and Masaharu Yoshioka. 2024. Overview of Benchmark Datasets and Methods for the Legal Information Extraction/Entailment Competition (COLIEE) 2024. In New Frontiers in Artificial Intelligence, Toyotaro Suzumura and Mayumi Bono (Eds.). Springer Nature Singapore, Singapore, 109–124. find
  7. Matthias Grabmair, Kevin D. Ashley, Ran Chen, Preethi Sureshkumar, Chen Wang, Eric Nyberg, and Vern R. Walker. 2015. Introducing LUIMA: an experiment in legal conceptual retrieval of vaccine injury decisions using a UIMA type system and tools. In Proceedings of the 15th International Conference on Artificial Intelligence and Law (San Diego, California) (ICAIL ’15). Association for Computing Machinery, New York, NY, USA, 69–78. https://doi.org/10.1145/2746090.2746096 ↗
  8. Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. 2024. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems 36 (2024). ↗
  9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. Proceedings of the International Conference on Learning Representations (ICLR) (2021). ↗
  10. Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2020. A dataset for statutory reasoning in tax law entailment and question answering. arXiv preprint arXiv:2005.05257 (2020). ↗
  11. Abe Bohan Hou, Orion Weller, Guanghui Qin, Eugene Yang, Dawn Lawrie, Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2024. CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generation. arXiv preprint arXiv:2406.17186 (2024). ↗
  12. Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky. 2023. Query expansion by prompting large language models. arXiv preprint arXiv:2305.03653 (2023). ↗
  13. Pengyue Jia, Yiding Liu, Xiangyu Zhao, Xiaopeng Li, Changying Hao, Shuaiqiang Wang, and Dawei Yin. 2023. Mill: Mutual verification with large language models for zero-shot query expansion. arXiv preprint arXiv:2310.19056 (2023). ↗
  14. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023). ↗
  15. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Vancouver, Canada.
  16. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2024. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A 382, 2270 (2024), 20230254. ↗
  17. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: a Benchmark for Question Answering Research. Transactions of the Association of Computational Linguistics (2019). ↗
  18. Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021. Mind the Gap: Assessing Temporal Generalization in Neural Language Models. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 29348–29363. https://proceedings.neurips.cc/paper_ files/paper/2021/file/f5bf0ba0a17ef18f9607774722f5698c-Paper.pdf ↗
  19. Yibin Lei, Yu Cao, Tianyi Zhou, Tao Shen, and Andrew Yates. 2024. Corpus-Steered Query Expansion with Large Language Models. arXiv preprint arXiv:2402.18031 (2024). ↗
  20. Jimmy Lin. 2019. The Neural Hype and Comparisons Against Weak Baselines. SIGIR Forum 52, 2 (jan 2019), 40–51. https://doi.org/10.1145/3308774.3308781 ↗
  21. Antoine Louis and Gerasimos Spanakis. 2022. A Statutory Article Retrieval Dataset in French. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Dublin, Ireland, 6789–6803. https://aclanthology.org/2022.acl-long.468 ↗
  22. Antoine Louis, Gijs van Dijck, and Gerasimos Spanakis. 2024. Interpretable long-form legal question answering with retrieval-augmented large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 22266–22275.
  23. LSC. 2021. Eviction Laws Database: Local Dataset. Prepared by the Center for Public Health Law Research at Temple University’s Beasley School of Law for Legal Services Corporation. https://www.lsc.gov/initiatives/effect-state-local-laws-evictions/lsc-eviction-laws-database. ↗
  24. Megan Ma, Aparna Sinha, Ankit Tandon, and Jennifer Richards. 2024. Generative AI Legal Landscape 2024. Technical Report. Technical report. find
  25. Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning, and Daniel E. Ho. 2024. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. arXiv:2405.20362 ↗
  26. Robert Mahari, Dominik Stammbach, Elliott Ash, and AlexSandy’ Pentland. 2023. LePaRD: A Large-Scale Dataset of Judges Citing Precedents. arXiv preprint arXiv:2311.09356 (2023). ↗
  27. Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. WWW’18 Open Challenge: Financial Opinion Mining and Question Answering. In Companion Proceedings of the The Web Conference 2018 (, Lyon, France,) (WWW ’18). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1941–1942. https://doi.org/10.1145/3184558.3192301 ↗
  28. Timothy McFarlin. 2023. A More Realistic Bar Exam Will Benefit Legal Education. The Bar Examiner 92, 2 (2023). ↗
  29. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. (2019). find
  30. Stephen E Robertson and K Sparck Jones. 1976. Relevance weighting of search terms. Journal of the American Society for Information science 27, 3 (1976), 129– 146. ↗
  31. Guilherme Moraes Rosa, Ruan Chaves Rodrigues, Roberto Lotufo, and Rodrigo Nogueira. 2021. Yes, bm25 is a strong baseline for legal case retrieval. arXiv preprint arXiv:2105.05686 (2021). ↗
  32. Jon Saad-Falcon, Daniel Y Fu, Simran Arora, Neel Guha, and Christopher Ré. 2024. Benchmarking and building long-context retrieval models with loco and m2-bert. arXiv preprint arXiv:2402.07440 (2024). ↗
  33. Jaromír Šavelka and Kevin D Ashley. 2022. Legal information retrieval for understanding statutory terms. Artificial Intelligence and Law (2022), 1–45. ↗
  34. Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S Siegel, Michael Tang, et al. 2024. BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval. arXiv preprint arXiv:2407.12883 (2024). ↗
  35. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https://openreview.net/forum?id=wCu6T5xFjeJ ↗
  36. Santosh T.y.s.s., Rashid Haddad, and Matthias Grabmair. 2024. ECtHR-PCR: A Dataset for Precedent Understanding and Prior Case Retrieval in the European Court of Human Rights. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (Eds.). ELRA and ICCL, Torino, Italia, 5473– 5483. https://aclanthology.org/2024.lrec-main.486 ↗
  37. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533 (2022). ↗
  38. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368 (2023). ↗
  39. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 5776–5788. https://proceedings.neurips.cc/paper_files/ paper/2020/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf ↗
  40. Xiaoyue Wang, Jianyou Wang, Weili Cao, Kaicheng Wang, Ramamohan Paturi, and Leon Bergen. 2024. BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives. arXiv preprint arXiv:2402.14151 (2024). ↗
  41. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP). ↗
  42. Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. 2021. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the eighteenth international conference on artificial intelligence and law. 159–168. find
  43. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. JEC-QA: a legal-domain question answering dataset. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 9701–9708. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2025-reasoning-focused-legal-retrieval-benchmark/