Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N18-2003.pdf
Summary
The paper asks whether coreference resolution systems rely on gender stereotypes about occupations. The authors build WinoBias, a Winograd-style challenge set of 3,160 sentences. Each sentence mentions two people by occupation, drawn from 40 jobs whose gender ratios come from US Department of Labor statistics. Every sentence appears in a male and a female pronoun version, and the pronoun's gender never decides the correct link. The sentences fall into two types: one requires world knowledge, the other can be solved through syntax.
Three systems were tested: a rule-based, a feature-rich, and an end-to-end neural one. All of them resolve pronouns better when the link matches the stereotype, with an average gap of 21.1 F1.
The authors trace part of this bias to the skewed gender balance of OntoNotes 5.0. To counter it, they combine three steps:
- anonymizing named entities,
- augmenting the training data with a gender-swapped copy,
- debiasing auxiliary resources such as embeddings and gender lists.
Together, these steps remove the measured bias at little cost to OntoNotes accuracy. The resulting diagnostic and mitigation recipe can be reused for other models.
Key points
- Systems tested: Stanford Deterministic Coreference System (Rule), Berkeley Coreference Resolution System (Feature) and UW End-to-end Neural Coreference Resolution System (E2E, Lee et al., 2017). The rule-based system shows the largest bias, with a 39.2 F1 pro/anti gap on Type 1 sentences.
- Training data imbalance: in OntoNotes 5.0, over 80% of entities with a mention headed by a gendered pronoun are male. Job titles also appear in male mentions at more than double the rate seen in female mentions.
- Gender-swapping rules: these were mined from Amazon Mechanical Turk rewrites of gendered spans in the OntoNotes development set. Named entities were first anonymized with an automatic NER tagger.
- Effect of debiasing on E2E: with anonymization, debiased GloVe vectors (Bolukbasi et al., 2016) and augmentation combined, the Type 1 gap falls from 26.6 to 1.1 F1. The Type 2 gap falls from 13.5 to 2.1 F1. OntoNotes F1 changes only from 67.7 to 66.3.
- Gender-reversed OntoNotes: on this version of the development set, E2E barely changes (66.4 vs. 65.9) and Feature loses about 1 F1. This suggests that strong in-domain cues partly mask the bias that WinoBias exposes.
Cited by
- MEGA: Multilingual Evaluation of Generative AI
- Recognition of They/Them as Singular Personal Pronouns in Coreference Resolution
- Analysis Methods in Neural Language Processing: A Survey
- Language (Technology) is Power: A Critical Survey of ``Bias'' in NLP
- Toward Gender-Inclusive Coreference Resolution
- Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution
- A Survey of Data Augmentation Approaches for NLP
- Bias and Fairness in Large Language Models: A Survey
- Evaluating Models' Local Decision Boundaries via Contrast Sets
- Gemini: A Family of Highly Capable Multimodal Models
- Improving alignment of dialogue agents via targeted human judgements
- Evaluating Large Language Models: A Comprehensive Survey
- Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual
- TrustLLM: Trustworthiness in Large Language Models
- OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
- Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
- Gender Bias in Neural Natural Language Processing
- A Survey on Bias and Fairness in Machine Learning
- CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
- Having Beer after Prayer? Measuring Cultural Bias in Large Language Models
- A Comprehensive Overview of Large Language Models
- Text and Code Embeddings by Contrastive Pre-Training
- LingMess: Linguistically Informed Multi Expert Scorers for Coreference Resolution
- Reducing Gender Bias in Abusive Language Detection
- BBQ: A hand-built bias benchmark for question answering
- Gender Bias in Coreference Resolution
- Multitask Prompted Training Enables Zero-Shot Task Generalization
- Gender Bias in Machine Translation
- On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
- The Woman Worked as a Babysitter: On Biases in Language Generation
- Prompting GPT-3 To Be Reliable
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
- Evaluating Gender Bias in Machine Translation
- Mitigating Gender Bias in Natural Language Processing: Literature Review
- A Multiscale Visualization of Attention in the Transformer Model
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns
- Measuring and Reducing Gendered Correlations in Pre-trained Models
- CorefQA: Coreference Resolution as Query-based Span Prediction
- Incremental Neural Coreference Resolution in Constant Memory
- AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference
- PAWS: Paraphrase Adversaries from Word Scrambling
- Learning Gender-Neutral Word Embeddings
- Gender Bias in Contextualized Word Embeddings
- PCR4ALL: A Comprehensive Evaluation Benchmark for Pronoun Coreference Resolution in English
- Can Large Language Models Transform Computational Social Science?
- Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology
- Representation Engineering: A Top-Down Approach to AI Transparency
Cite
@inproceedings{zhao2018gender,
title = {{Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods}},
author = {Zhao, Jieyu and Wang, Tianlu and Yatskar, Mark and Ordonez, Vicente and Chang, Kai-Wei},
booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
year = {2018},
doi = {10.18653/v1/n18-2003},
eprint = {1804.06876},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/n18-2003}
}References
- Shane Bergsma and Dekang Lin. 2006. Bootstrapping path-based pronoun resolution. In ACL. find
- Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In NIPS. find our card
- Toon Calders, Faisal Kamiran, and Mykola Pech-enizkiy. 2009. Building classifiers with independency constraints. In Data mining workshops, 2009. ICDMW’09. IEEE international conference on. IEEE, pages 13–18. find
- Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356(6334):183–186. find
- Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Empirical Methods on Natural Language Processing. find our card
- Michael Collins. 2003. Head-driven statistical models for natural language parsing. Computational linguistics 29(4):589–637. 10 We do not evaluate the Rule system as it cannot be train for anonymized input. 11 Their dataset also includes gender neutral pronouns and examples containing one job title instead of two. find our card
- Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington. find our card
- Karthik Raghunathan, Heeyoung Lee, Sudarshan Rangarajan, Nathanael Chambers, Mihai Surdeanu, Dan Jurafsky, and Christopher Manning. 2010. A multi-pass sieve for coreference resolution. In EMNLP. pages 492–501. find our card
- Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubrama-nian. 2015. Certifying and removing disparate impact. pages 259–268. find
- Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: The winograd schema challenge. In EMNLP. pages 777–789. find our card
- Karen Ross and Cynthia Carter. 2011. Women and news: A long and winding road. Media, Culture & Society 33(8):1148–1165. find
- Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2014. Randomized significance tests in machine translation. In WMT@ ACL. pages 266–274. find
- Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In NAACL. find our card
- Graeme Hirst. 1981. Anaphora in natural language understanding. Berlin Springer Verlag . find
- Latanya Sweeney. 2013. Discrimination in online ad delivery. Queue 11(3):10. find
- Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. 2012. Fairness-aware classifier with prejudice remover regularizer. Machine Learning and Knowledge Discovery in Databases pages 35–50. find
- Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Nianwen Xue, Martha Palmer, Jena D Hwang, Claire Bonial, et al. 2012. Ontonotes release 5.0 . find
- Toshihiro Kamishima, Shotaro Akaho, and Jun Sakuma. 2011. Fairness-aware learning through regularization approach. In Data Mining Workshops (ICDMW), 2011 IEEE 11th International Conference on. IEEE, pages 643–650. find
- Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In EMNLP. find our card
- Matthew Kay, Cynthia Matuszek, and Sean A Munson. 2015. Unequal representation and gender stereotypes in image search results for occupations. In Human Factors in Computing Systems. ACM, pages 3819–3828. find
- Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. In Advances in Neural Information Processing Systems. pages 4069–4079. find
- Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360 . ↗
- Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In EMNLP. find our card
- Ishan Misra, C Lawrence Zitnick, Margaret Mitchell, and Ross Girshick. 2016. Seeing through the human reporting bias: Visual classifiers from noisy human-centric labels. In CVPR. pages 2930–2939. find
- Haoruo Peng, Kai-Wei Chang, and Dan Roth. 2015a. A joint framework for coreference resolution and mention head detection. In CoNLL. page 10. find
- Haoruo Peng, Daniel Khashabi, and Dan Roth. 2015b. Solving hard coreference problems. In NAACL. find
- Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In ACL. find our card