📄 Classifying Interpretive Canons at the Sentence Level: A Benchmark from the German Federal Constitutional Court

Summary

Studies of judicial reasoning usually tag argument types per paragraph. That shows whether an argument occurs but not exactly where. Ringe moves the task to single sentences, using the four classical interpretive canons (grammatical, systematic, historical, objective-teleological) as formulated by Larenz in Savigny's tradition. He breaks the notion of a "reading" into explicit tests: the statement must be abstract, a genuine assertion, tied to a specific statutory provision with determinate content, and advanced by the court itself. These tests become separate subtasks. One expert annotated a dataset of German Federal Constitutional Court decisions, recording reasons for edge cases. Fifteen decisions are labelled exhaustively, and extra material was added for rare classes. Four LLMs from three families were tested with expert-written prompts and with GEPA-optimized prompts. Mean F1 across seven binary subtasks lies between 70.4 and 79.2. Grammatical arguments are usually easiest to detect and systematic ones hardest. GEPA does not reliably beat the expert prompts. The benchmark gives a reproducible, fine-grained baseline for large-scale empirical legal research.

Key points

• The models evaluated are DS-V4-Pro, DS-V4-Flash, Gemini-3.5-FL and MiniMax-M3, each run with an expert prompt and a GEPA prompt.
• There are seven binary subtasks, and mean F1 across them ranges from 70.4 to 79.2:
  • two reading criteria: nicht_abstrakt and nicht_selbst_aufgestellt;
  • an argument gate;
  • four canon labels: wortlaut, systematik, geschichte, zweck.
• konkretes_gesetz (the statutory reference) is handled differently. It is a citation-extraction target scored by per-document set-overlap F1, and its span-level score is much lower.
• Systematic interpretation scores lowest in six of eight model–prompt configurations, with a mean of 58.4. Historical (70.2) and teleological (70.9) are roughly tied.
• All gold labels come from one annotator, so no inter-annotator agreement can be reported. The data covers one court in one language, and the test sets are small, which makes the confidence intervals wide.
• The code is hosted on GitHub, the datasets are published via Hugging Face, and the evaluation runs are logged in Weights & Biases. An AI agent reproduced the reported scores and datasets from the method descriptions alone, without the code.

Interesting ideas

• I1. Labels relative to a reference point. A sentence's canon depends on the reference point. The same sentence counts as grammatical with respect to one provision and systematic with respect to another, so the label belongs to a sentence–anchor pair rather than to the sentence alone.
• I2. Fuzzy concepts as a conjunction of tests. The vague idea of a "reading" is turned into several narrow, checkable conditions, each scored separately, with a gate in front of the finer labels. This gives an error profile per condition instead of one opaque judgement.
• I3. Error analysis that names the surface proxy. False positives were traced to concrete shortcuts the models took: the word "Wortlaut", any parallel citation, or any section number. The systematic canon fails both ways because its proxy is unreliable.
• I4. Labelling every sentence of a multi-sentence argument. This exposed that models miss continuation and conclusion sentences that lack cue words, which hurts recall for gradually built arguments.
• I5. Reproduction by an agent from the paper alone. An independent agent rebuilt both the data and the scores from the written methods, which tests how complete the documentation is.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): Argument and stance relations in prag could be stored as edges toward an explicit target, not as properties of a clause. Two frames could then interpret the same clause differently without conflict.
• I2 (frames): Speech-act and assertion detection could use the same tests: is the claim generic or about a specific case, is it asserted or merely quoted, and is it the speaker's own claim or attributed to someone else. Each test would be a separate, explainable feature.
• I3 (learn): For each classifier or repair rule, list its likely surface proxy. Then build negative controls where the cue is present but the phenomenon is absent, and the reverse.
• I4 (world): Track discourse units that span sentences, so a concluding sentence inherits its role from the chain it closes rather than needing its own lexical trigger.
• I5 (learn): Evaluation pipelines could be checked the same way. Regenerate silver data and metrics from the spec only, and treat any mismatch as a documentation or determinism bug.