📄 LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain

Summary

RAG tools for law are spreading quickly, but existing legal benchmarks such as LegalBench only test how well an LLM reasons when it is handed the correct passage. None of them check whether a system can find that passage in a large document collection. LegalBench-RAG addresses this gap. The authors took the clauses that LegalBench supplies as context and located each one in its original source document. Each answer is recorded as an exact character span. They then turned the source annotations into natural-language queries. The result is 6,858 expert-annotated query–span pairs over a corpus of more than 79M characters, drawn from four datasets. A smaller LegalBench-RAG-mini is also provided for quick experiments.

In baseline tests, a Recursive Text Character Splitter beat naive chunking. Adding a general-purpose Cohere reranker unexpectedly lowered both precision and recall. The benchmark matters because legal answers need short, exact passages: short passages fit context limits, reduce hallucination, and let users check citations.

Key points

• The benchmark is built from four LegalBench sources:
  • ContractNLI (946 queries)
  • CUAD (4,042 queries)
  • MAUD (1,676 queries)
  • PrivacyQA (194 queries)
• Ground truth for each query is a document file plus one or more character ranges, not a document ID or a loose chunk.
• Queries follow the template "Consider (document description); (interrogative)". The description identifies the document, and the interrogative is mapped from the annotation category.
• Recall@k and Precision@k were measured for k from 1 to 64, and the four datasets were weighted equally in the aggregate.
• PrivacyQA turned out easiest: with RTCS and no reranker it reached about 14.4% Precision@1 and 84.2% Recall@64. MAUD was hardest, at about 2.6% Precision@1 and 28.3% Recall@64.

Interesting ideas

• I1. Convert a reader benchmark into a retrieval benchmark. The authors traced each gold context passage back to its location in the full source documents. Any QA dataset with provided contexts can be repurposed this way, without new annotation work.
• I2. Character-span ground truth. Labelling answers as exact offsets, instead of document or chunk IDs, makes evaluation independent of how a system splits text. It also rewards precise snippets that fit in a context window and can be cited.
• I3. Self-contained queries from templates. Each query pairs a unique document description with a category question. This makes queries answerable across a whole corpus without hand-writing them, which is useful for building retrieval sets from classification-style annotations.
• I4. Equal weighting per sub-dataset. Averaging each source dataset equally, regardless of its size, keeps a large component like CUAD from dominating the score. This is a sensible default for any composite benchmark.
• I5. Test general-purpose components on specialized text. The finding that an off-the-shelf reranker hurt results on legal text argues for checking each pipeline stage on in-domain data rather than assuming it helps. This likely applies to other specialized domains as well.