中文句子级性别无偏数据集构建及预训练语言模型的性别偏度评估

Summary

Chinese NLP had no dedicated resource for measuring or reducing gender bias. Existing bias test sets are mostly English, hand-built, small and tied to specific tasks. The authors address this gap by automatically mining naturally occurring, gender-neutral sentences from Chinese newspaper text.

They start from 16 pairs of gendered address/kinship terms and extract matching sentences from the DCC corpus. Rule-based filtering, word segmentation and NER remove irrelevant sentences and those containing personal names. Linguist annotators then keep only sentences whose context fits either gender equally well. The result is SlguSet, about 20,000 sentences.

Bias is measured with masked language modelling. The gendered keyword is hidden, and the log ratio of the male and female fill-in probabilities is computed. The absolute values are averaged over all sentences. Applied to five Chinese pre-trained models, the metric shows that every model is biased to some degree.

Qualitative analysis shows typical stereotypes. Men are linked to work, leadership and money. Women are linked to children, food and appearance. The dataset and metric give a low-cost, reusable benchmark for Chinese.

Key points

• Data and annotation:

  • Source: 2018–2019 print-media text from the DCC (a 10-billion-character corpus).
  • Keywords: 16 gender term pairs, e.g. 他/她, 男/女, 爸爸/妈妈.
  • Annotators: three linguistics master's students, who kept a sentence only with 100% agreement.
  • Exclusions: sentences with co-referring gender terms or biological-sex cues (e.g., pregnancy).

• Character coverage: the dataset uses 4,220 distinct characters, about 19.97% of BERT-Chinese's 21,128-entry vocabulary.

• Models and overall bias scores:

  Model	Bias score
  BERT-wwm-ext	1.4360 (most biased)
  RoBERTa-wwm-ext	1.4280
  BERT-wwm	1.3829
  BERT-base (Chinese)	1.3113
  ELECTRA-base	1.2676 (least biased)

  • Per-sentence bias scores are strongly correlated across the five models.

• BERT-base breakdown: with a threshold of |Bias_c| > 0.3:

  • 6,024 sentences lean male;
  • 6,743 lean female;
  • 7,223 count as neutral.

• Topic and case-study findings:

  • TF-IDF/TextRank topic words are "child" for female-leaning sentences and "work" or "a (person)" for male-leaning ones.
  • In the 男童/女童 (boy/girl) case study, 78 sentences lean "boy" and 158 lean "girl". Boy contexts involve mischief and accidents; girl contexts involve protection and abuse.