📄 GRUFF: LLM Pronoun Fidelity, Reasoning, and Biases in German

Summary

Pronoun fidelity asks whether a model, after learning which pronoun a person uses, keeps using that pronoun later even when sentences about other people come in between. Earlier work on this task covered only English, which has almost no gender agreement. The authors build GRUFF, a large German dataset that adapts the English RUFF design. In GRUFF, articles, nouns and pronouns must all agree. The dataset combines 60 occupation–participant pairs, four noun-agreement systems (masculine, feminine, the De-e system, the Sternchen form), four pronoun sets (er, sie, xier, en) and all four grammatical cases. Up to five distractor sentences can be inserted. Encoder-only and decoder-only models are scored by forced choice over their probabilities. Agreement with er and sie is nearly perfect even without an introductory context. Neopronouns are handled only when the context states them, and en remains weak. Distractors still hurt performance, but encoder models cope better than in English. Occupational stereotypes show weak correlation across cases and across most models. The work shows that fairness results from English do not carry over directly to languages with grammatical gender.

Key points

• GRUFF contains 240 occupation-specific task templates (60 occupations × 4 cases) and 40 generic context templates (10 × 4 cases).
• Each setting with introductory context is subsampled into 3 sets of 2,880 instances.
• Each instance has three parts:
  • an introduction sentence that ties an entity to a pronoun;
  • 0–5 distractor sentences about a second entity, which are either explicit or continue that entity by pronoun only;
  • a task sentence with a pronoun gap.
• German possessives inflect for both possessor and possessed. To keep endings uniform, all genitive templates use feminine possessed nouns, which give a consistent -e ending.
• Models evaluated include GBERT-base and GBERT-large, SauerkrautLM-8B and SauerkrautLM-70B, and Llama-70B. Fidelity is best for er, sie and xier and very low for en. There is no gap between er and sie.
• The authors measured stereotype rankings with Spearman correlation.
  • Friseur, Planer and Küchenchef are over-resolved to ihn; Berater, Kassierer and Börsenmakler are over-resolved to sie.
  • Rankings correlate only between closely related models, such as the two GBERT sizes or SauerkrautLM-70B and Llama-70B. Even SauerkrautLM-8B and SauerkrautLM-70B barely correlate.

Interesting ideas

• I1. Build a coreference test from three template slots (introduce, distract, probe) so that the number of intervening sentences about another entity can be varied exactly from 0 to 5.
• I2. Score bias by over-resolution, meaning how often a system picks a pronoun that contradicts explicit grammatical cues, rather than raw accuracy.
• I3. Compare a no-context condition with an explicit-introduction condition to separate plain grammatical agreement from true tracking of a stated referent.
• I4. Fix a confounding morphological variable when building templates, as the authors did with feminine possessed nouns in the genitive, so that scoring stays clean.
• I5. Check whether bias rankings agree across grammatical cases and across models before treating a stereotype as stable.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): This would give the coreference module a graded stress test, showing how pronoun resolution decays as distractor sentences about other entities pile up between mention and pronoun.
• I2 (learn): This could serve as a negative control in evaluation, flagging cases where a learned prior overrides an explicit cue in the tree or morphology.
• I3 (world): Running world-of-text questions with and without the introducing sentence would show whether an answer comes from the entity's recorded state or from defaults.
• I4 (answer): Test generators for Mova's readers could pin irrelevant inflection or number choices so that a failure points to the skill being tested.
• I5 (learn): When several witness parsers or LLM teachers agree on silver data, measuring the rank correlation of their systematic errors would reveal shared biases that agreement alone would hide.