Elamur Research
☰
home / papers / 2009 / Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art

Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art

What it is about

The paper asks why published coreference scores differ so much between the MUC and ACE data sets. It uses the RECONCILE platform: a classifier with about 61 features, an averaged perceptron, and single-link clustering. Three subtasks are measured by giving the resolver more information. Gold named entities help only by about 0.5 to 3 points. Perfect coreference elements (CEs) help a great deal, and the authors estimate that better CE detection could add 10 to 20 MUC F points. Knowing anaphoricity also helps. The second part splits anaphors into nine resolution classes: proper names and common nouns, each with exact string match, partial match or no match to the antecedent, and three pronoun types. The average MUC-RC scores are 0.89 for proper names with exact match, 0.21 for proper names with no match, 0.18 for common nouns with no match, 0.80 for gendered third-person pronouns and 0.48 for ungendered ones. A corpus's predicted score from its class mix correlates with the observed score at 0.913 without MUC-6 (0.731 with it).

Ideas

For Mova

The code is coref/src/eval.rs (classes and per-class MUC-RC) and coref/src/mention.rs (CE extraction). The chapter is languages/en/35-reference-and-coreference.md. The measure is the per-class score on the Mova dev split. Negative control: shuffling antecedents within a class must lower that class's score, and a class that does not move under the shuffle is not measuring anything.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P09-1074.pdf

Summary

Published results for noun phrase coreference resolution vary widely, with F-scores ranging from roughly 60 to over 90. It is unclear whether these gaps come from better methods, easier corpora or more lenient evaluation settings. The authors separate these factors using six standard benchmarks: MUC-6, MUC-7 and four ACE newswire sets. Their system is RECONCILE ACL09, a pairwise-classification resolver built on the Reconcile platform. First, they compare how MUC and ACE define coreference elements (CEs). They then measure how much end-to-end scores change when the system receives gold named entities, gold anaphoricity labels, or exactly the annotated CEs. Oracle anaphoricity and oracle CE information produce large gains, while perfect NER adds little. This shows that some common evaluation assumptions make the task unrealistically easy. Second, they score the resolver separately on different resolution classes of anaphora. From these scores they build a Coreference Performance Prediction (CPP) measure, which estimates performance on an unseen corpus from its mix of classes. The study provides baselines with clearly stated assumptions and a practical way to judge how hard a new dataset will be.

Key points

  • The MUC and ACE task definitions differ in important ways. ACE includes singletons, relative pronouns and gerunds, and limits CEs to 7 semantic classes. MUC has no semantic restriction, annotates only coreferent CEs, and allows nested nouns.
  • RECONCILE ACL09 pipeline: OpenNLP, the Berkeley parser and the Stanford CRF NER for preprocessing. It uses 61 features (lexical, proximity, grammatical, semantic), the Averaged Perceptron and single-link clustering. Its heuristics recover about 90% of the annotated CEs.
  • Typical reported scores are 85–90% on ACE 2004/2005 but only 60–70% on MUC-6/7. This partly reflects the different task definitions, not just system quality.
  • A new MUC-RC scorer evaluates each resolution class on its own. For the other classes it takes the correct (gold) clustering.
  • For each corpus, CPP is computed from the average per-class scores on the other five corpora. Predicted and observed scores correlate at 0.731 across all six sets and 0.913 without MUC-6. On MUC-6 the resolver beats the prediction (0.67 observed vs. 0.59 predicted), which the authors attribute to MUC-6's annotation of nested nouns and features originally tuned on that dataset.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{stoyanov2009conundrums,
  title = {{Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art}},
  author = {Stoyanov, Veselin and Gilbert, Nathan and Cardie, Claire and Riloff, Ellen},
  booktitle = {Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP},
  year = {2009},
  url = {https://aclanthology.org/P09-1074/}
}

References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Bagga and B. Baldwin. 1998. Algorithms for Scoring Coreference Chains. In In Linguistic Coreference Workshop at LREC 1998. find our card
  2. Baldridge, J. 2005. The OpenNLP project. http://opennlp.sourceforge.net/. ↗
  3. D. Bean and E. Riloff. 2004. Unsupervised Learning of Contextual Role Knowledge for Coreference Resolution. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL 2004). find
  4. Eric Bengtson and Dan Roth. 2008. Understanding the Value of Features for Coreference Resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 294–303. Association for Computational Linguistics. find our card
  5. Alexandra Birch, Miles Osborne, and Philipp Koehn. 2008. Predicting Success in Machine Translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 745–754. Association for Computational Linguistics. find
  6. Donna Byron. 2001. The Uncommon Denominator: A Proposal for Consistent Reporting of Pronoun Resolution Results. Computational Linguistics, 27(4):569–578. find
  7. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating Typed Dependency Parses from Phrase Structure Parses. In LREC. find our card
  8. J. Finkel, S. Dingare, H. Nguyen, M. Nissim, and C. Manning. 2004. Exploiting Context for Biomedical Entity Recognition: From Syntax to the Web. In Joint Workshop on Natural Language Processing in Biomedicine and its Applications at COLING 2004. find
  9. Yoav Freund and Robert E. Schapire. 1999. Large Margin Classification Using the Perceptron Algorithm. In Machine Learning, pages 277–296. find
  10. Megumi Kameyama. 1997. Recognizing Referential Links: An Information Extraction Perspective. In Workshop On Operational Factors In Practical Robust Anaphora Resolution For Unrestricted Texts. find
  11. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A Mention-Synchronous Coreference Resolution Algorithm Based on the Bell Tree. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics. find
  12. X. Luo. 2005. On Coreference Resolution Performance Metrics. In Proceedings of the 2005 Human Language Technology Conference / Conference on Empirical Methods in Natural Language Processing. find our card
  13. Xiaoqiang Luo. 2007. Coreference or Not: A Twin Model for Coreference Resolution. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL 2007). find
  14. A. McCallum and B. Wellner. 2004. Conditional Models of Identity Uncertainty with Application to Noun Coreference. In 18th Annual Conference on Neural Information Processing Systems. find
  15. Ruslan Mitkov. 2002. Anaphora Resolution. Longman, London. find
  16. MUC-6. 1995. Coreference Task Definition. In Proceedings of the Sixth Message Understanding Conference (MUC-6), pages 335–344. find
  17. MUC-7. 1997. Coreference Task Definition. In Proceedings of the Seventh Message Understanding Conference (MUC-7). find
  18. V. Ng and C. Cardie. 2002a. Identifying Anaphoric and Non-Anaphoric Noun Phrases to Improve Coreference Resolution. In Proceedings of the 19th International Conference on Computational Linguistics (COLING 2002). find
  19. V. Ng and C. Cardie. 2002b. Improving Machine Learning Approaches to Coreference Resolution. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. find our card
  20. NIST. 2004. The ACE Evaluation Plan. find
  21. S. Petrov and D. Klein. 2007. Improved Inference for Unlexicalized Parsing. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL 2007). find
  22. W. Soon, H. Ng, and D. Lim. 2001. A Machine Learning Approach to Coreference of Noun Phrases. Computational Linguistics, 27(4):521–541. find
  23. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, Ellen Riloff, David Buttler, and David Hysom. 2009. Reconcile: A Coreference Resolution Research Platform. Computer Science Technical Report, Cornell University, Ithaca, NY. find
  24. Kees van Deemter and Rodger Kibble. 2000. On Coreferring: Coreference in MUC and Related Annotation Schemes. Computational Linguistics, 26(4):629–637. find our card
  25. M. Vilain, J. Burger, J. Aberdeen, D. Connolly, and L. Hirschman. 1995. A Model-Theoretic Coreference Scoring Theme. In Proceedings of the Sixth Message Understanding Conference (MUC-6). find
  26. Xiaofeng Yang, Guodong Zhou, Jian Su, and Chew Lim Tan. 2003. Coreference Resolution Using Competition Learning Approach. In ACL ’03: Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, pages 176–183. find