📄 The Birth of Bias: A case study on the evolution of gender bias in an English language model

Summary

Most work on gender bias in language models detects or removes it after training. This paper asks a prior question: how does a model become biased while it learns? The authors chose a small, fully transparent setup, an LSTM language model trained on an English Wikipedia corpus. They had complete access to the training data and to the parameters at every training step. This let them follow how the model's encoding of gender changes over time, which data patterns drive those changes, and how internal states connect to biased behaviour when the model is applied to semantic textual similarity. Gender representation turns out to be dynamic and passes through distinct training phases. Over training, gender information becomes increasingly concentrated, or local, in the input embeddings, so debiasing those embeddings reduces downstream bias. Watching the training dynamics also exposed an asymmetry between how female and male gender are encoded. Simple mitigation methods that ignore this asymmetry could introduce new unwanted biases. Only the abstract is available here, because the paper is a non-archival publication. The extracted header also shows a different title, which looks like a PDF glitch.

Key points

• Model and data: an LSTM language model trained on an English Wikipedia corpus, with every training-step checkpoint and the full dataset inspectable.
• Downstream bias was measured on semantic textual similarity (STS) and related to the model's internal representations.
• Gender encoding goes through several identifiable phases during training rather than growing steadily.
• Gender information becomes more localized in the input embeddings over training, so debiasing at that layer effectively lowers STS bias.
• Female and male gender are represented asymmetrically in the input embeddings. The authors discuss extending the approach to Transformers, other languages and other biases.

Interesting ideas

• I1. Study a property across all training checkpoints instead of only in the final model; the timeline shows phases that a single snapshot hides.
• I2. Link an internal measurement directly to a downstream outcome, so you can tell whether a representation-level fix actually changes behaviour.
• I3. First check whether information is localized before choosing where to intervene; a fix at the input layer only works if the signal is concentrated there.
• I4. Check that a phenomenon is symmetric before applying a symmetric correction; naive debiasing of an asymmetric encoding can create new bias.
• I5. Use a small, fully transparent model as a lab before scaling claims to larger architectures.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Save checkpoints of the perceptron parser and gparse and track error classes over training, to see when a systematic mistake appears and which data introduced it.
• I2 (learn): Pair each repair rule or silver-data change with a downstream test score, such as QA on fables, so an internal improvement is only accepted if the answers also improve.
• I3 (world): Before adding gender or animacy features to entities, check whether that signal sits in the lexicon or ontology layer or is spread across frames, and place the correction at the layer where it is concentrated.
• I4 (frames): In coreference, test pronoun resolution separately for he/she and for each antecedent type, and treat each direction on its own if their error profiles differ.
• I5 (learn): Run bias and negative-control experiments on the small perceptron parser with fully known training data first, then check whether the findings carry over to gparse.