Elamur Research
☰
home / papers / 2019 / Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology

Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1906.04571

Summary

Counterfactual data augmentation (CDA) reduces gender bias by adding copies of sentences with the gender flipped. It was developed for English, where swapping words like he/she is enough. In languages with grammatical gender agreement, the same swap produces broken sentences, because determiners, adjectives and sometimes verbs must change along with the noun. The authors propose a four-stage pipeline. It analyzes a sentence (parsing, morphological tagging and lemmatization, based on Universal Dependencies (UD) annotations), switches the gender of an animate noun, infers how the remaining morpho-syntactic tags must adjust, and reinflects the lemmata into new surface forms. The core component is a delexicalized Markov random field defined over the dependency tree, with either linear or neural edge potentials. It is trained without any paired masculine/feminine sentences, since no such corpus exists. In an intrinsic evaluation on Spanish and Hebrew, it reaches high tag-level F1 and form-level accuracy. In an extrinsic evaluation across four languages, language models trained on the augmented corpora show substantially less gender stereotyping while largely keeping grammaticality. This matters for debiasing NLP in gender-marking languages, such as Slavic languages including Ukrainian.

Key points

  • The intrinsic results cover Spanish and Hebrew. Tag-level F1 is about 82–83% and 72–73% respectively; the abstract and introduction report slightly different figures. Form-level accuracy is 90% and 87%.
  • In the extrinsic evaluation, the authors trained BPE-RNNLM, a language model without a fixed vocabulary introduced by Mielke and Eisner (2018), on French, Hebrew, Italian and Spanish corpora. Each language had three versions: the original corpus, a naïve-swap CDA corpus and the MRF-based CDA corpus.
  • Gender stereotyping fell by a factor of 2.5 on average. The smallest drop was 1.2× (Italian) and the largest 5.0× (Spanish). Naïve swapping helped for only some of the languages.
  • Grammaticality was usually better than with naïve swapping, and sometimes higher than in the original corpus. Hebrew is the exception, which matches its weaker intrinsic accuracy.
  • A noun counts as animate if WordNet lists person as one of its hypernyms. The partition function is computed with belief propagation, and unary factors are used only to fix the intervened tag.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

157 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 149 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1906.04571 · arXiv 1906.04571

@misc{zmigrod2019counterfactual,
  title = {{Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology}},
  author = {Zmigrod, Ran and Mielke, Sabrina J. and Wallach, Hanna and Cotterell, Ryan},
  year = {2019},
  doi = {10.48550/arxiv.1906.04571},
  eprint = {1906.04571},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1906.04571}
}

References

33 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016,… find our card
  2. Francis Bond and Kyonghee Paik. 2012. A survey of WordNets and their licenses. In Proceedings of the 6th Global WordNet Conference (GWC 2012), Matsue. 64–71. find
  3. Jennifer Coates. 1987. Women, Men and Language: A Sociolinguistic Account of Sex Differences in Language. Longman. find
  4. Greville G. Corbett. 1991. Gender. Cambridge University Press. find
  5. Greville G. Corbett. 2012. Features. Cambridge University Press. find
  6. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. find
  7. CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 1–30, Vancouver. Association for Computational Linguistics. find
  8. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. find
  9. The SIGMORPHON 2016 shared task— morphological reinflection. In Proceedings of the 2016 Meeting of SIGMORPHON, Berlin, Germany. Association for Computational Linguistics. find
  10. Kate Crawford. 2013. The hidden biases in big data. find
  11. Maria De-Arteaga, Alexey Romanov, Hanna M. Wallach, Jennifer T. Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Cem Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference… find
  12. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. find
  13. Timothy Dozat and Christopher D. Manning. 2016. Deep biaffine attention for neural dependency parsing. CoRR, abs/1611.01734. find our card
  14. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  15. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2017. Word embeddings quantify 100 years of gender and ethnic stereotypes. CoRR, abs/1711.08412. — [arXiv:1711.08412](https://arxiv.org/abs/1711.08412) find
  16. Nizar Habash, Houda Bouamor, and Christine Chung. 2019. Automatic gender identification and reinflection in arabic. In Proceedings of the 1st ACL Workshop on Gender Bias for Natural Language Processing, Florence, Italy. find
  17. Labiba Jahan, Geeticka Chauhan, and Mark Finlayson. 2018. A new approach to animacy detection. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1–12. Association for Computational Linguistics. find
  18. Diederik P. Kingma and Jimmy Ba. 2014. A method for stochastic optimization. abs/1412.6980. find
  19. Adam: CoRR, Daphne Koller and Nir Friedman. 2009. Probabilistic graphical models: Principles and techniques. MIT Press. find
  20. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2018. Gender bias in neural natural language processing. CoRR, abs/1807.11714. find our card
  21. Sabrina J. Mielke and Jason Eisner. 2018. Spell once, summon anywhere: A two-level open-vocabulary language model. CoRR, abs/1804.08205. — [arXiv:1804.08205](https://arxiv.org/abs/1804.08205) find
  22. Thomas Müller, Ryan Cotterell, Alexander Fraser, and Hinrich Schütze. 2015. Joint lemmatization and morphological tagging with lemming. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2268–2274. Association for Computational Linguistics. find
  23. Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Katya Aplonova, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mi… find
  24. Judea Pearl. 1988. Probabilistic reasoning in intelligent systems: Networks of plausible inference. Morgan Kaufmann Publishers. find
  25. Lawrence R. Rabiner and Biing-Hwang Juang. 1986. An introduction to hidden Markov models. IEEE ASSP Magazine, 3(1):4–16. find
  26. Louis B. Rall. 1981. Automatic Differentiation: Techniques and Applications, volume 120 of Lecture Notes in Computer Science. Springer. find
  27. Rachel Rudinger, Chandler May, and Benjamin Van Durme. 2017. Social bias in elicited natural language inferences. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 74–79. Association for Computational Linguistics. find our card
  28. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–1… find our card
  29. Adam Sutton, Thomas Lansdall-Welfare, and Nello Cristianini. 2018. Biased embeddings from wild data: Measuring, understanding and removing. CoRR, abs/1806.06301. — [arXiv:1806.06301](https://arxiv.org/abs/1806.06301) find
  30. Shijie Wu, Pamela Shapiro, and Ryan Cotterell. 2018. Hard non-monotonic attention for character-level transduction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4425–4438. Association for Computational Linguistics. find
  31. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1… find our card
  32. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. pages 2979–2989. find our card
  33. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. pages 15–20. find our card