📄 When Rules Learn: A Self-Evolving Agent for Legal Case Retrieval

Summary

Lexical retrieval with BM25 remains surprisingly strong for Chinese legal case retrieval. On LeCaRD-v2, it beats both general and legal-domain dense embedders. The authors therefore improve BM25 instead of replacing it. Their approach rewrites queries with an LLM that follows explicit, human-readable rewriting rules.

Writing such rules by hand requires legal expertise, so the paper lets an LLM agent evolve the rule set itself. The agent works in a closed loop with an automatic evaluation environment and has three possible moves:

• create or edit a rule,
• test a chosen combination of active rules by having an external model rewrite development queries and scoring recall,
• discard rules that underperform.

All decisions are prompt-driven, and no model weights are updated.

With a strong core model (gpt-oss-120b), the evolved rules outperform hand-crafted rules and greedy rule selection. Analyses show that success depends on two abilities: reusing earlier experimental results when planning new tests, and judging when to prune rules. The work points to a training-free, interpretable route for adapting retrieval in data-scarce legal settings.

Key points

• Baseline comparison: On LeCaRD-v2, BM25 reaches an average recall of 58.69. This is higher than bge-m3 (50.04), SAILER_zh (46.21) and ReaKase-8B (57.07). At Recall@50, the legal-domain models trail BM25 by roughly 7.6–19.1%.
• Agent design: The action space is rule creation, experiment planning and rule elimination. Newly generated rules must not resemble rules that were previously removed.
• Consistency of gpt-oss-120b: Across five runs, gpt-oss-120b was the only core LLM that beat the greedy strategy at every recall cutoff.
• Rule set size: Qwen3-4B-Thinking ended with about 37.8±3.9 active rules, despite being instructed to keep roughly 6. gpt-oss-120b kept about 5.8±1.0 but sometimes pruned useful rules too early.
• Non-reasoning models: Models such as Qwen3-30B-A3B-Instruct produced unusable rules, and Magistral-Small-2509 also generated garbled rules. The study therefore relies on four reasoning LLMs.