Elamur Research
☰
home / papers / 2022 / Recognition of They/Them as Singular Personal Pronouns in Coreference Resolution

Recognition of They/Them as Singular Personal Pronouns in Coreference Resolution

What it is about

The paper tests whether coreference systems resolve singular personal "they" (a named person) as a singular referent. It builds WinoNB schemas: about 4,077 templates adapted by hand from Winogender, WinoBias, WSC and DPR, in which one referent is a group of people and the other a named individual, filled with 15 names. Each pair differs in a small edit so that the correct antecedent flips. Five systems are tested: Clark and Manning, two variants of the Lee et al. end-to-end model, BERT and SpanBERT. On the "all" row of Table 2, accuracy on singular personal "they" is 0.008 to 0.145 for the five systems, against 0.846 to 0.988 for plural "they". For the singular generic case ("someone") accuracy is 0.011 to 0.618. The three OntoNotes-only models resolve "they" almost never as singular. BERT-base does best on the personal case (about 3 times the average), which the authors link to exposure to singular "they" in pre-training. A hand check of 100 sampled OntoNotes sentences found no singular personal "they".

Ideas

For Mova

The sieve is coref/src/sieve.rs, with mention features in coref/src/mention.rs. The gold split is by number and by the pronoun form. The chapter is languages/en/12-pronouns-determiners-numerals.md (the they-pronoun and number of pronouns) and languages/en/35-reference-and-coreference.md. Measure: the WinoNB-style accuracy for the sieve, split by the three cases, run twice. Keep the set separate from any release test set.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.naacl-main.250.pdf

Summary

Coreference resolution systems tend to treat "they" as plural, which disadvantages people, especially nonbinary individuals, who use they/them as personal pronouns. The authors built WinoNB schemas, a controlled benchmark that tests whether a model can link "they" to a single named person rather than to a group. They hand-edited Winograd-style items from Winogender, WinoBias, WSC and DPR. In each item one candidate referent became a group (often an occupation in plural form) and the other a named individual, so the correct reading is singular in one variant and plural in the other. A second version replaces the name with "someone" to probe generic singular "they." Five off-the-shelf systems were tested, and all of them strongly preferred the plural reading or left the pronoun unresolved. Generic "they" was handled considerably better. BERT, likely exposed to modern usage through Wikipedia during pre-training, performed best. The work shows a measurable representational harm and offers a resource for building more inclusive models. It can reveal bias but cannot prove its absence, and it does not cover neopronouns.

Key points

  • Benchmark construction: WinoNB contains 4077 templates derived from Winogender, WinoBias, WSC and DPR. Each template is filled with 15 names: 10 common AMAB/AFAB names and 5 gender-neutral ones. A fluent English speaker resolved a check sample of 100 templates with 96% accuracy.
  • Systems evaluated: C&M, End-to-End (Lee et al. 2018), C&M++ (Hugging Face), BERT and SpanBERT. All were trained or fine-tuned on OntoNotes 5.0 / CoNLL-2012. A sample of 100 OntoNotes sentences with they/them contained no singular personal use.
  • Singular vs. plural gap: On average, models were 94.8% less likely to resolve singular personal "they" correctly than plural "they."
  • Model differences: BERT-base reached about 3× the average singular-personal accuracy. However, it left 5.15% of those cases unresolved, which is more than double the average rate. SpanBERT scored lower, possibly because of stronger optimization for OntoNotes.
  • Generic "they" and name gender: Generic "they" (with "someone") averaged 31.2% accuracy, 6.4× higher than personal "they." Errors were more frequent with AMAB names; for example, BERT-base made about 7.5% more errors on Winogender-based items with those names.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.naacl-main.250

@inproceedings{baumler2022recognition,
  title = {{Recognition of They/Them as Singular Personal Pronouns in Coreference Resolution}},
  author = {Baumler, Connor and Rudinger, Rachel},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.250},
  url = {https://doi.org/10.18653/v1/2022.naacl-main.250}
}

References

18 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Al-locative versus representational harms in machine learning. In SIGCIS Conference. find
  2. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics. ↗ our card
  3. Yang Trista Cao and Hal Daumé III. 2020. Toward gender-inclusive coreference resolution. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568–4595, Online. Association for Computational Linguistics. find our card
  4. Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256–2262, Austin, Texas. Association for Computational Linguistics. find our card
  5. Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff Phillips, and Kai-Wei Chang. 2021. Harms of gender exclusivity and challenges in non-binary representation in language technologies. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1968–1994, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  6. Julie Foertsch and Morton Ann Gernsbacher. 1997. In search of gender neutrality: Is singular they a cognitively efficient substitute for generic he? Psychological science, 8(2):106–111. find
  7. Milton Friedman. 1937. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200):675–701. find
  8. Milton Friedman. 1940. A comparison of alternative tests of significance for the problem of m rankings. The Annals of Mathematical Statistics, 11(1):86–92. find
  9. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. find our card
  10. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China. Association for Computational Linguistics. ↗ our card
  11. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card
  12. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
  13. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguistics. find our card
  14. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777–789, Jeju Island, Korea. Association for Computational Linguistics. find our card
  15. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–14, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card
  16. Thomas Wolf. 2017. State-of-the-art neural coreference resolution for chatbots. Blog post. find
  17. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics. find our card
  18. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27. find our card