Elamur Research
☰
home / papers / 2019 / The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution

The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution

What it is about

KnowRef is a set of 8,724 natural Winograd-like passages: two person noun phrases, then one connective (but, because, although, when, so …), then a pronoun that agrees with both («Paul helped Lionel hide when he was pursued by the authorities»). It is mined by filters: sentence length, exactly one connective cluster, exactly two person NPs before it and none repeated after it, no pronoun before it. Training labels come free: keep sentences whose two names differ in gender, read the answer off the pronoun's gender (86% right on a check), then replace one name so that the gender cue disappears. The test set (Wikipedia and subtitles) is labelled by humans with high agreement. Rule-based, feature-based and neural resolvers all do far worse than people, and a «consistency» test (swap the two names; a model using context should keep pointing at the same position) shows they mostly follow gender and number. Training on name-swapped copies ("antecedent switching") helps, also on GAP.

Ideas

For Mova

languages/en/35-reference-and-coreference.md §2 (salience), §8 (algorithms), §10 (the resolver inside the frame reader). Code: coref/src/sieve.rs (pronoun), prag/src/frames.rs (Ctx), world/src/events.rs (verb statistics from books). Measures: precision on the harvested held-out set, he/she/they precision on GUM dev, the swap-consistency gate.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1386.pdf

Summary

Standard coreference benchmarks such as the CoNLL 2011/2012 tasks can often be solved through surface signals like the gender or number of candidate antecedents. The Winograd Schema Challenge avoids these cues but is small and hand-written. Emami et al. address both problems with KnowRef, a large benchmark built from naturally occurring text. Each passage contains an ambiguous pronoun and two same-type named entities that appear before it, so only context and common-sense knowledge can identify the correct referent.

The authors also introduce a consistency metric. It swaps the two candidate names and checks whether a model's prediction changes accordingly. Rule-based, statistical and neural resolvers all fall clearly short of humans on KnowRef, and many largely ignore context.

The swapping idea also serves as a data-augmentation method called antecedent switching. It improves fine-tuned BERT on KnowRef and produces state-of-the-art results on GAP while balancing performance across feminine and masculine pronouns. The work matters because it exposes shortcut reliance in coreference models and offers a simple way to reduce that reliance, along with gender bias.

Key points

  • KnowRef contains 8,724 human-labeled, Winograd-like passages, much larger than the WSC's 273 instances. The corpus, scraping code and experiment code are released at github.com/aemami1/KnowRef.
  • Under the consistency test, the deterministic rule-based sieve (Raghunathan et al., 2010) never changes its answer when the names are swapped (0%). It depends on a gender/number dictionary rather than context.
  • Training the end-to-end neural model (Lee et al.) on KnowRef raises its consistency, which suggests the corpus pushes models toward contextual reasoning.
  • Antecedent switching doubles the training data and has these effects on fine-tuned BERT:
    • Accuracy on KnowRef rises by 10 points, to 71%.
    • Consistency reaches 89%.
    • Inconsistent resolutions drop from 31% to 11%.
  • On GAP, BERT with augmentation reaches an F1 of 71.1 (+1.9) with a female:male F1 ratio of 1.00. This beats the Parallelism+URL baseline (70.6).

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1386 · arXiv 1811.01747

@inproceedings{emami2019knowref,
  title = {{The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution}},
  author = {Emami, Ali and Trichelair, Paul and Trischler, Adam and Suleman, Kaheer and Schulz, Hannes and Cheung, Jackie Chi Kit},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1386},
  eprint = {1811.01747},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1386}
}

References

33 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dan Bailey, Amelia Harrison, Yuliya Lierler, Vladimir Lifschitz, and Julian Michael. 2015. The winograd schema challenge and reasoning about correlation. In Working Notes of the Symposium on Logical Formalizations of Commonsense Reasoning. find
  2. David Bean and Ellen Riloff. 2004. Unsupervised learning of contextual role knowledge for coreference resolution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004. find
  3. Shane Bergsma and Dekang Lin. 2006. Bootstrapping path-based pronoun resolution. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. find
  4. Kevin Clark and Christopher D Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. find our card
  5. Kevin Clark and Christopher D Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. find our card
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  7. George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie Strassel, and Ralph M Weischedel. 2004. The automatic content extraction (ace) program-tasks, data, and evaluation. In LREC. find
  8. Greg Durrett, David Hall, and Dan Klein. 2013. Decentralized entity-level modeling for coreference resolution. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. find
  9. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. find our card
  10. Ralph Grishman and Beth Sundheim. 1996. Message understanding conference-6: A brief history. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics, volume 1. find
  11. Jerry R Hobbs. 1977. SIGART Bulletin. Pronoun resolution. find
  12. ACM Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the conll-2011 shared task. In Proceedings of the fifteenth conference on computational natural language learning: Shared task. find our card
  13. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. find our card
  14. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find our card
  15. Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. find
  16. Quan Liu, Hui Jiang, Andrew Evdokimov, Zhen-Hua Ling, Xiaodan Zhu, Si Wei, and Yu Hu. 2016. Probabilistic reasoning via deep learning: Neural association models. arXiv preprint arXiv:1603.07704. ↗
  17. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association of Computational Linguistics. find our card
  18. Karthik Raghunathan, Heeyoung Lee, Sudarshan Rangarajan, Nathanael Chambers, Mihai Surdeanu, Dan Jurafsky, and Christopher Manning. 2010. A multi-pass sieve for coreference resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find our card
  19. Altaf Rahman and Vincent Ng. 2009. Supervised models for coreference resolution. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  20. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: the winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. find our card
  21. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find our card
  22. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology. find
  23. Andrew McCallum and Ben Wellner. 2005. Conditional models of identity uncertainty with application to noun coreference. In Advances in neural information processing systems. find
  24. Paul Trichelair, Ali Emami, Jackie Chi Kit Cheung, Adam Trischler, Kaheer Suleman, and Fernando Diaz. 2018. On the evaluation of common-sense reasoning in natural language understanding. The NeurIPS Workshop on Critiquing and Correcting Trends in Machine Learning. find
  25. JF McCarthy. 1995. Using decision trees for coreference resolution. In Proc. 14th International Joint Conf. on Artificial Intelligence. find
  26. Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847. ↗
  27. Thomas S Morton. 2000. Coreference for nlp applications. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics. find
  28. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the gap: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. ↗ our card
  29. Haoruo Peng, Daniel Khashabi, and Dan Roth. 2015. Solving hard coreference problems. Urbana. find
  30. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL-Shared Task. find our card
  31. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task. find our card
  32. Sam Wiseman, Alexander M Rush, and Stuart M Shieber. 2016. Learning global features for coreference resolution. In Proceedings of NAACL-HLT. find our card
  33. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In NAACL-HLT. find our card