Elamur Research⌕
☰
home / papers / 2026 / CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval

CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval

Paper · 2026 · Akshith Reddy Putta, Jacob Devasier and Chengkai Li

Summary

Most automated fact-checking work verifies general-knowledge claims against fixed sources such as Wikipedia. Law poses harder problems. Ordinary people describe rulings in everyday language, while the evidence sits in technical judicial text, and a legal claim can stop being true once its precedent is overturned. The authors present CaseFacts, a benchmark that pairs informal legal statements with U.S. Supreme Court cases. Each claim carries one of three labels: SUPPORTED, REFUTED, or OVERRULED.

They built it from expert-written Oyez case summaries. An LLM generated claims, and a second, different LLM acted as a judge. It filtered out claims that were not entailed by the case, as well as contradictory or redundant ones. A semantic-similarity heuristic narrowed down cross-case pairs, which made it feasible to find overrulings without comparing every pair.

In the experiments, current LLMs struggle with the task. A GPT-4o with open web search did worse than the same model without retrieval, because search returned noisy, non-authoritative cases. Embedding models fine-tuned on the training split improved retrieval recall. The resource targets time-aware legal verification grounded in evidence.

Key points

  • The dataset holds 6,294 claims drawn from 3,299 SCOTUS cases on Oyez (as of July 2025). It includes a human-annotated test set and is released at github.com/idirlab/CaseFacts.
  • The pipeline used open-weight models from different families to reduce self-preference bias: Qwen3-Next-80B-A3B-Thinking generated the claims and gemma-3-27b-it validated them.
  • Filtering reduced the claim pool in stages: 10,471 initial claims, then 8,987 after the factuality check, then 7,583 after removing contradictions and overlaps within the same case.
  • A standard NLI model (textattack/bert-base-uncased-MNLI) could not detect legal overrulings or confirmations. Instead, the authors used similarity windows: 0.70–0.95 for contradictions and 0.80–0.97 for overlaps, avoiding more than 25 million pairwise LLM checks.
  • The authors note several limitations. The corpus uses summaries rather than full opinions, the hard negatives are synthetic, and coverage is limited to the U.S. Supreme Court, with no statutes, state law, or other countries.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at ArlingtonArlington · United States6 works · 1,516 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 worksPolitical Science and International Relations · 182 works

🧭 Topics 3Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksInformation retrieval & search · 473 worksLegal AI · 134 works

Cite

Cite as Akshith Reddy Putta, Jacob Devasier and Chengkai Li. 2026. CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.785

@inproceedings{putta2026casefacts,
  title = {{CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval}},
  author = {Akshith Reddy Putta, Jacob Devasier and Chengkai Li},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.785},
  url = {https://doi.org/10.18653/v1/2026.acl-long.785}
}

References

25 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S Ajay Mukund and K. S. Easwarakumar. 2025. Optimizing Legal Text Summarization Through Dynamic Retrieval-Augmented Generation and Domain-Specific Adaptation. Symmetry, 17(5). ↗
  2. Mubashara Akhtar, Oana Cocarascu, and Elena Simperl. 2022. PubHealthTab: A Public Health Table-based Dataset for Evidence-based Fact Checking. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1–16, Seattle, United States. Association for Computational Linguistics. find
  3. Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. FEVEROUS: Fact extraction and VERification over unstructured and structured information. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). ↗
  4. Avishek Anand, Procheta Sen, Sourav Saha, Manisha Verma, and Mandar Mitra. 2023. Explainable Information Retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 3448–3451, New York, NY, USA. Association for Computing Machinery. find
  5. Tobias Braun, Mark Rothermel, Marcus Rohrbach, and Anna Rohrbach. 2024. DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts. arXiv preprint arXiv:2412.10510. ↗
  6. Szu-Ju Chen, Jing Jin, Sheng-Lun Wei, Chien-Hung Chen, and Hsin-Hsi Chen. 2025. Retrieving the Right Law: Enhancing Legal Search with Style Translation. In Proceedings of the 48th International ACM SI-GIR Conference on Research and Development in Information Retrieval, SIGIR ’25, page 2951–2955, New York, NY, USA. Association for Computing Machinery. find
  7. Jacob Devasier, Akshith Reddy Putta, Rishabh Mediratta, and Chengkai Li. 2025. Task-Oriented Automatic Fact-Checking with Frame-Semantics. In Findings of the Association for Computational Linguistics: ACL 2025, pages 13825–13842, Vienna, Austria. Association for Computational Linguistics. find
  8. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 Technical Report. Preprint, arXiv:2503.19786. ↗
  9. Randy Goebel, Yoshinobu Kano, Mi-Young Kim, Juliano Rabelo, Ken Satoh, and Masaharu Yoshioka. 2024. Overview of Benchmark Datasets and Methods for the Legal Information Extraction/Entailment Competition (COLIEE) 2024. In New Frontiers in Artificial Intelligence, pages 109–124, Singapore. Springer Nature Singapore. find
  10. Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, and 1 others. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in neural information processing systems, 36:44123–44279.
  11. Yinghao Hu, Leilei Gan, Wenyi Xiao, Kun Kuang, and Fei Wu. 2025. Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4410– 4427, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  12. Larissa Mori, Carlos Sousa de Oliveira, Yuehwern Yih, and Mario Ventresca. 2025. Assessing the performance gap between lexical and semantic models for information retrieval with formulaic legal language. In Proceedings of the Twentieth International Conference on Artificial Intelligence and Law, pages 114– 128. find
  13. Mihai Nadǎş, Laura Dioşan, and Andreea Tomescu. 2025. Synthetic Data Generation Using Large Language Models: Advances in Text and Code. IEEE Access, 13:134615–134633. ↗
  14. Akshith Reddy Putta, Jacob Devasier, and Chengkai Li. 2025. ClaimCheck: Real-Time Fact-Checking with Small Language Models. Preprint, arXiv:2510.01226. ↗
  15. Qwen Team. 2025. Qwen3 Technical Report. Preprint, arXiv:2505.09388. ↗
  16. Markus Reuter, Tobias Lingenberg, Ruta Liepina, Francesca Lagioia, Marco Lippi, Giovanni Sartor, Andrea Passerini, and Burcu Sayin. 2025. Towards Reliable Retrieval in RAG Systems for Large Legal Datasets. In Proceedings of the Natural Legal Language Processing Workshop 2025, pages 17–30, Suzhou, China. Association for Computational Linguistics. ↗
  17. Michael Sejr Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. 2023. AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. ↗
  18. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33:5776–5788. find
  19. Yuxia Wang, Minghan Wang, Hasan Iqbal, Georgi N. Georgiev, Jiahui Geng, Iryna Gurevych, and Preslav Nakov. 2025. OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 11399–11421, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  20. Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2025. Self-Preference Bias in LLM-as-a-Judge. Preprint, arXiv:2410.21819. ↗
  21. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641–649.
  22. Jingze Zhang, Jiahe Qian, Yiliang Zhou, and Yifan Peng. 2025a. Enhancing Health Fact-Checking with LLM-Generated Synthetic Data. Preprint, arXiv:2508.20525. ↗
  23. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025b. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. Preprint, arXiv:2506.05176. ↗
  24. Rongwen Zhao and Jeffrey Flanigan. 2025. SYN-THVERIFY: Enhancing Zero-Shot Claim Verification through Step-by-Step Synthetic Data Generation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 3257–3274, Vienna, Austria. Association for Computational Linguistics. find
  25. Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. 2021. When does pre-training help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the eighteenth international conference on artificial intelligence and law, pages 159–168. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/putta-2026-casefacts-benchmark-legal-fact-checking/