📄 WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case

Summary

Winogender Schemas are a widely used benchmark for gender bias in coreference resolution. The authors audit it and find problems that make its measurements unreliable. Pronoun forms in different grammatical cases are counted as if they were interchangeable, and their counts are very uneven. Some paired templates break the structural rules they are supposed to follow, and the data contains typos. About a sixth of the templates also fail under singular they because of verb agreement.

The authors rebuild the dataset as WinoPron. It has 360 templates, balanced across nominative, accusative and possessive. It adds the neopronoun set xe/xem/xyr and single-entity control sentences. The data was validated with automatic structural checks and human annotation.

They evaluate LingMess, CAW-coref, SpanBERT and five sizes of FLAN-T5. All of them score lower on WinoPron than on the original data, and accusative pronouns are the hardest case for every model. They also propose a bias measure that does not reduce pronouns to a he/she contrast. With it, they show that bias can differ between surface forms of the same pronoun set. The broader lesson is that claims about bias are only as good as the data behind them.

Key points

• Case imbalance. The original dataset had 89 nominative, 4 accusative and 27 possessive templates. WinoPron has 120 of each, so 360 templates × 4 pronoun sets give 1,440 evaluation sentences.

• Singular they. 17% of the original templates did not work with singular they (e.g. "they were" vs. "he was"). Templates were rewritten so that any pronoun fits, for example by using past-tense verbs and avoiding was/were.

• Validation. Both template authors rated 100% of instances as grammatical with a unique, correct antecedent. An independent annotator agreed on grammaticality for all of them and on coreference for 98.2%.

• Performance drops on WinoPron:

  Model	Winogender Schemas	WinoPron
  LingMess	85.5	64.4
  CAW-coref	81.3	67.3
  SpanBERT-base	71.8	61.6
  SpanBERT-large	82.0	70.1

• Hard pronouns. Supervised systems struggle badly with singular they and xe, while some FLAN-T5 sizes handle them surprisingly well. Data and code are at github.com/uds-lsv/winopron.

Interesting ideas

• I1. Break down evaluation by the surface form and grammatical case of a pronoun, not just by pronoun set, because averaging hides that object-position pronouns fail much more often.
• I2. Pair each two-entity test sentence with a simpler single-entity version, so a failure can be traced to the model being unable to do the task at all rather than to confusion between antecedents.
• I3. Encode the design rules of a test set as programmatic checks, such as paired sentences being identical up to the pronoun and containing no stray extra pronouns.
• I4. Build minimal pairs where only the sentence ending decides the antecedent, so that "always pick the first entity" or "always pick the second" heuristics score at chance.
• I5. Write templates so pronouns can be swapped without breaking agreement, and discard any variant that becomes ambiguous (e.g. plural readings of they, or "someone's diagnosis" readings of possessives).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): report coreference accuracy separately for he/him/his-type slots so weaknesses tied to accusative or possessive positions show up.
• I2 (answer): add single-entity control questions next to multi-entity fable QA items, to tell whether a reader fails at reference itself or at choosing between characters.
• I3 (learn): run similar validators over self-play or LLM-generated test items before they enter evaluation or silver data.
• I4 (frames): use such pairs as negative controls showing the resolver relies on frames and world knowledge rather than position.
• I5 (back): when generating pronoun-swapped variants from the meaning graph, regenerate verb agreement and flag outputs where the pronoun could now attach to more than one entity.