Elamur Research
☰
home / papers / 2013 / Easy Victories and Uphill Battles in Coreference Resolution

Easy Victories and Uphill Battles in Coreference Resolution

What it is about

The paper builds a learning-based coreference system, the Berkeley system, that uses only a small set of homogeneous feature templates over shallow mention properties (head match, exact string match, and syntactic and discourse features) instead of many hand-built heuristics. A log-linear mention-ranking model chooses one antecedent for each mention or starts a new cluster. Mentions are extracted by simple rules (named-entity chunks, PRP pronouns, nominal NP projections), so the system has many spurious mentions that the model must reject. Evaluation uses the CoNLL 2011 data with system mentions, scored with MUC, B3 and CEAF-e and their average, the CoNLL metric. The system beats the Stanford system (the CoNLL 2011 winner) by 3.5 points and the IMS system by 1.9 points on the CoNLL metric. The authors report that the features win "easy victories" on syntax and discourse, but semantic features (shallow lexical semantics from the literature) add nothing with system mentions; they help only with gold mentions, which they explain by the number of spurious mentions that weak evidence cannot overcome. I read the abstract, the introduction and the setup; I did not read the later feature tables.

Ideas

For Mova

Code: coref/src/mention.rs (three mention types from UD PRON, PROPN, NP heads), coref/src/sieve.rs (pairwise features), coref/src/eval.rs (MUC, B3, CEAF, CoNLL average). Chapter: languages/en/35-reference-and-coreference.md. Measure: the CoNLL average on a dev split with system mentions and with gold mentions, run twice; the gap between the two is the mention-detection share.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/D13-1203.pdf

Summary

Coreference resolution systems usually rely on many hand-built rules and mixed feature types. Each rule targets one phenomenon, such as pronoun agreement, discourse salience or anaphoricity, which makes these systems hard to understand and extend. Durrett and Klein show that a simpler design can do better. Their log-linear mention-ranking (mention-synchronous) model makes one choice per mention: link it to a single earlier mention, or start a new cluster with it. Training uses a softmax-margin objective that weights error types differently and sums over all antecedent choices consistent with the gold clusters. The features come from a few uniform templates over shallow surface cues and are combined with mention type, which produces a very large lexicalized feature space. On the CoNLL 2011 shared task data (OntoNotes), these features model syntax and discourse better than the hand-engineered ones they replace. Shallow semantic features, however, help only when gold mentions are given. With automatically extracted mentions, their weak signal cannot overcome the many spurious candidates. The released Berkeley Coreference Resolution System beats the Stanford and IMS systems and gives the field a simple, strong baseline.

Key points

  • On the CoNLL 2011 test set, the FINAL system scores 60.13 on the CoNLL metric (the average of MUC, B³ and CEAFe). That is 3.5 points above Stanford (Lee et al., 2011; 56.65) and 1.9 above IMS (Björkelund and Farkas, 2012; 58.26).
  • On the development set, the SURFACE feature set alone reaches 60.06, already ahead of both baselines. FINAL adds features such as syntactic ancestry, gender/number, containment and speaker match, and reaches 61.58.
  • Mention detection aims for maximal recall without filtering rules. Spurious mentions are treated as singletons during training, and singletons are removed before scoring.
  • The loss weights for false anaphor, false new and wrong link are 0.1, 3.0 and 1.0. This pushes the model toward making links despite the many singletons. Training uses AdaGrad with L1 regularization, all mentions are resolved in a single pass, and no coreference arcs are pruned.
  • The system was released publicly (nlp.cs.berkeley.edu), and all scores use version 5 of the official CoNLL scorer.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d13-1203

@inproceedings{durrett2013easy,
  title = {{Easy Victories and Uphill Battles in Coreference Resolution}},
  author = {Durrett, Greg and Klein, Dan},
  booktitle = {Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  doi = {10.18653/v1/d13-1203},
  url = {https://doi.org/10.18653/v1/d13-1203}
}

References

37 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for Scoring Coreference Chains. In Proceedings of the Conference on Language Resources and Evaluation Workshop on Linguistics Coreference. Avg. F 1 F 1 47.07 46.66 49.00 50.09 57.67 58.13 60.06 61.58 45.17 45.35 47.79 56.65 58.26 60.13 find our card
  2. Mohit Bansal and Dan Klein. 2012. Coreference Semantics from Web Features. In Proceedings of the Association for Computational Linguistics. find
  3. Eric Bengtson and Dan Roth. 2008. Understanding the Value of Features for Coreference Resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  4. Shane Bergsma and Dekang Lin. 2006. Bootstrapping Path-Based Pronoun Resolution. In Proceedings of the Conference on Computational Linguistics and the Association for Computational Linguistics. find
  5. Anders Björkelund and Richárd Farkas. 2012. Data-driven Multilingual Coreference Resolution using Resolver Stacking. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Proceedings and Conference on Computational Natural Language Learning - Shared Task. find
  6. Anders Björkelund and Pierre Nugues. 2011. Exploring Lexicalized Features for Coreference Resolution. In Proceedings of the Conference on Computational Natural Language Learning: Shared Task. find
  7. Jie Cai and Michael Strube. 2010. Evaluation Metrics for End-to-End Coreference Resolution Systems. In Proceedings of the Special Interest Group on Discourse and Dialogue. find
  8. Chen Chen and Vincent Ng. 2012. Combining the Best of Two Worlds: A Hybrid Approach to Multilingual Coreference Resolution. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Proceedings and Conference on Computational Natural Language Learning - Shared Task. find
  9. Pascal Denis and Jason Baldridge. 2008. Specialized Models and Ranking for Coreference Resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  10. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12:2121–2159, July. find
  11. Greg Durrett, David Hall, and Dan Klein. 2013. Decentralized Entity-Level Modeling for Coreference Resolution. In Proceedings of the Association for Computational Linguistics. find
  12. Eraldo Rezende Fernandes, Cı́cero Nogueira dos Santos, and Ruy Luiz Milidiú. 2012. Latent Structure Perceptron with Feature Induction for Unrestricted Coreference Resolution. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Proceedings and Conference on Computational Natural Language Learning - Shared Task. find our card
  13. Kevin Gimpel and Noah A. Smith. 2010. Softmax-Margin CRFs: Training Log-Linear Models with Cost Functions. In Proceedings of the North American Chapter for the Association for Computational Linguistics. find
  14. David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2007. English Gigaword Third Edition. Linguistic Data Consortium, Catalog Number LDC2007T07. find
  15. Barbara J. Grosz, Scott Weinstein, and Aravind K. Joshi. 1995. Centering: A Framework for Modeling the Local Coherence of Discourse. Computational Linguistics, 21(2):203–225, June. find our card
  16. Aria Haghighi and Dan Klein. 2009. Simple Coreference Resolution with Rich Syntactic and Semantic Features. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  17. Aria Haghighi and Dan Klein. 2010. Coreference Resolution in a Modular, Entity-Centered Model. In Proceedings of the North American Chapter of the Association for Computational Linguistics. find
  18. Iris Hendrickx and Walter Daelemans, 2007. Adding Semantic Information: Unsupervised Clusters for Coreference Resolution. find
  19. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% Solution. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Short Papers. find
  20. Emmanuel Lassalle and Pascal Denis. 2013. Improving Pairwise Coreference Models Through Feature Space Hierarchy Learning. In Proceedings of the Association for Computational Linguistics. find
  21. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task. In Proceedings of the Conference on Computational Natural Language Learning: Shared Task. find our card
  22. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A Mention-Synchronous Coreference Resolution Algorithm Based on the Bell Tree. In Proceedings of the Association for Computational Linguistics. find
  23. Xiaoqiang Luo. 2005. On Coreference Resolution Performance Metrics. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  24. Sebastian Martschat, Jie Cai, Samuel Broscheit, Éva Mújdricza-Maydt, and Michael Strube. 2012. A Multigraph Model for Coreference Resolution. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Proceedings and Conference on Computational Natural Language Learning - Shared Task. find
  25. Simone Paolo Ponzetto and Michael Strube. 2006. Exploiting Semantic Role Labeling, WordNet and Wikipedia for Coreference Resolution. In Proceedings of the North American Chapter of the Association of Computational Linguistics. find
  26. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes. In Proceedings of the Conference on Computational Natural Language Learning: Shared Task. find our card
  27. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task. find our card
  28. Altaf Rahman and Vincent Ng. 2009. Supervised Models for Coreference Resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  29. Altaf Rahman and Vincent Ng. 2011a. Coreference Resolution with World Knowledge. In Proceedings of the Association for Computational Linguistics: Human Language Technologies. find
  30. Altaf Rahman and Vincent Ng. 2011b. Narrowing the Modeling Gap: A Cluster-Ranking Approach to Coreference Resolution. Journal of Artificial Intelligence Research, 40(1):469–521, January. find
  31. Marta Recasens, Matthew Can, and Daniel Jurafsky. 2013a. Same Referent, Different Words: Unsupervised Mining of Opaque Coreferent Mentions. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  32. Marta Recasens, Marie-Catherine de Marneffe, and Christopher Potts. 2013b. The Life and Death of Discourse Entities: Identifying Singleton Mentions. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  33. Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong Lim. 2001. A Machine Learning Approach to Coreference Resolution of Noun Phrases. Computational Linguistics, 27(4):521–544, December. find our card
  34. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art. In Proceedings of the Association for Computational Linguistics. find our card
  35. Veselin Stoyanov, Claire Cardie, Nathan Gilbert, Ellen Riloff, David Buttler, and David Hysom. 2010. Coreference Resolution with Reconcile. In Proceedings of the Association for Computational Linguistics: Short Papers. find
  36. Yannick Versley, Simone Paolo Ponzetto, Massimo Poesio, Vladimir Eidelman, Alan Jern, Jason Smith, Xiaofeng Yang, and Alessandro Moschitti. 2008. BART: A Modular Toolkit for Coreference Resolution. In Proceedings of the Association for Computational Linguistics: Demo Session. find
  37. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A Model-Theoretic Coreference Scoring Scheme. In Proceedings of the Conference on Message Understanding. find our card