📄 BenGER Platform: A Collaborative Web Platform for End-to-End Benchmarking of German Legal Tasks

Summary

Benchmarking LLMs on legal reasoning usually involves several disconnected tools. Experts supply material, engineers turn it into benchmark items, annotations are collected in a separate tool, models are run through one-off scripts, and evaluation code is rewritten for each study. This fragmentation hides decisions from legal experts and makes results hard to reproduce. The authors present BenGER (Benchmark for German Law), an open-source browser platform that handles the whole cycle in one system: task and reference-solution authoring, collaborative human annotation, batch runs of arbitrary LLMs, and scoring with a mix of metric families. It supports several organizations working together on one deployment, with each organization's data kept separate and access controlled by role. It can also give annotators LLM-generated feedback anchored in the reference answers. This is a short demo paper, so it reports no experiments. Its contribution is the system design and a live public instance. The aim is to let non-technical legal experts, public bodies and NGOs build and audit benchmarks without relying on engineers.

Key points

• The platform follows a six-stage workflow:
  • task creation with reference solutions
  • annotation
  • optional formative feedback
  • model execution
  • evaluation
  • analysis and export
• Supported task formats are free-text reasoning, multiple choice and span annotation. Metric families are lexical, semantic, factual, classification and LLM-as-a-judge.
• The stack is a Next.js (TypeScript) frontend and a FastAPI (Python) backend on PostgreSQL. Redis and Celery workers run model jobs and evaluations in the background. Deployment uses Docker Compose or Kubernetes.
• Each organization's data is kept separate in storage, and permissions depend on roles such as administrator, contributor and annotator. API keys can be set per user or per project, so institutions can use their own credentials.
• The authors position BenGER against LabelStudio, Doccano, Lawnotation and DeepWrite, which each cover only part of the pipeline. A public instance runs at what-a-benger.net, and the code is promised as free software on acceptance (ICAIL 2026 demo).

Interesting ideas

• I1. BenGER treats an evaluation run as a saved, shareable object made of four parts: task definition, reference solution, model configuration and chosen metrics. Such a run can be audited and repeated later instead of living inside a notebook.
• I2. The same task is scored with several kinds of metrics side by side, from surface overlap to factual checks to a judge model. No single score can then dominate the conclusion.
• I3. Annotator feedback is grounded in the reference answer and modeled on the German "Repetitor" tutor. It names missing reasoning steps and common mistakes rather than just marking an answer wrong.
• I4. Human baselines are collected inside the same system as model runs, and agreement and consistency signals are tracked. Noisy or inconsistently interpreted annotations then do not silently skew conclusions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): every evaluation, including its negative controls, could be saved as a versioned bundle of data slice, model build and metric set, so regressions are rerun against exactly the same setup.
• I2 (answer): answers on fable QA and school tests could be scored both by string match and by a fact-level check against the world graph, revealing answers that are right for the wrong reasons.
• I3 (answer): the maths word-problem solver could compare its derivation with a reference solution and report which step is missing, keeping the explanation chain-based rather than giving a bare verdict.
• I4 (learn): per-item agreement indicators among witness parsers could be stored next to the silver data, so low-agreement items are flagged before they enter training.