📄 On Measuring Gender Bias in Translation of Gender-neutral Pronouns

Summary

The paper tackles a gap in fairness research: there was no systematic way to check whether Korean-to-English MT systems invent a gender when translating Korean third-person pronouns that carry none. The authors build an equity evaluation corpus (EEC) from template sentences of the form "S/he is [xx]". The slot is filled with sentiment expressions or occupation names, and two further dimensions are varied: the pronoun used (informal kyay versus formal ku salam) and the presence or absence of a politeness suffix. The result is 4,236 test sentences grouped into seven subsets.

They also define a measure, the translation gender bias index (TGBI). For each subset, it scores how often outputs are female, male or neutral, and it rewards neutral or balanced renderings. They apply it to three commercial translators. Google Translator scores highest overall and Kakao lowest, while Papago appears to handle occupations differently.

The work matters because it gives a reusable, mechanically scorable resource and metric for comparing systems on gender-neutrality preservation.

Key points

• The lexicon has 1,059 entries:
  • 324 sentiment items (124 positive, 200 negative), taken from the Korean Sentiment Word Dictionary of Kunsan National University, which contains 14,843 items.
  • 735 occupation terms, taken from an official Korean government employment website.
• Each entry is combined with two pronouns (kyay, ku salam) and with politeness on or off. This gives 4,236 utterances in seven subsets covering (in)formality, (im)politeness, polarity and occupation.
• Words were filtered to avoid confounds:
  • Sentiment items were kept only if three native speakers fully agreed on their polarity.
  • Sentiment items touching on appearance, wealth, sexual orientation, disability, education or status were removed.
  • Gender-marked jobs (e.g., ballerino, woman diver) and hateful terms were dropped.
• The metric works in two steps:
  • Per subset, P_S = √(p_w·p_m) + p_n, where p_w, p_m and p_n are the shares of female, male and neutral translations. It peaks when all outputs are neutral and bottoms out when all are one gender.
  • TGBI is the average of P_S across the seven subsets.
• The systems evaluated were Google Translator, Naver Papago and Kakao Translator. The corpus and code are released at github.com/nolongerprejudice/tgbi.

Interesting ideas

• I1. Template-based probes from a gender-neutral source language. Starting from a language whose pronouns lack gender turns any gendered output into measurable bias. The same probe design could be used for other language pairs where the source is less marked than the target.
• I2. A metric with a built-in ideal. Combining a geometric mean of the female and male shares with the neutral share means both balanced guessing and neutral wording earn credit, while one-sided output is penalized. This bounded score is easy to adapt for other binary attributes that a system should not hallucinate.
• I3. Varying register as a separate axis. Treating formality and politeness as their own dimensions shows whether bias shifts with style rather than content. This is especially useful for languages with rich honorific or register systems.
• I4. Filtering lexicon items for hidden confounds. Removing words tied to appearance, wealth, disability and similar traits keeps the test focused on gender alone. This cleaning step makes any bias-probing word list more trustworthy.
• I5. Context-sensitive sanity checks. Sentences where only one gender is physiologically possible, such as being on a period, test whether a system can tell when guessing a gender is appropriate and when it is required. This complements neutrality tests in other evaluation suites.