Elamur Research⌕
☰
home / papers / 2020 / Conundrums in Entity Coreference Resolution: Making Sense of the State of the Art

Conundrums in Entity Coreference Resolution: Making Sense of the State of the Art

Summary

Neural entity coreference resolvers have achieved large score gains with fairly simple architectures: a mention-ranking model over contextualized embeddings with learned span representations. It has remained unclear what these gains actually come from. Lu and Ng address this with a systematic empirical analysis of five variants of three state-of-the-art resolvers: the Stanford neural resolver, Lee et al. (2018) with GloVe or GloVe+ELMo, and Joshi et al. (2019) with SpanBERT-base or SpanBERT-large. They evaluate them on ACE 2005, OntoNotes and KBP. For ACE and KBP, which annotate singletons, they add a jointly trained supervised mention detector to the span-based models, and they tune all hyperparameters on development data.

The resolvers rank in the same order on every dataset, and better mention detection always goes with better coreference. Newer models benefit from both stronger mention detection and better span representations. Input perturbations show that the resolvers depend mostly on the mentions themselves rather than their context. The work clarifies what the field has gained and points to mention detection, anaphoricity and entity type detection as remaining bottlenecks.

Key points

  • Datasets: ACE 2005 (Bansal and Klein split), OntoNotes (standard split, no singletons) and KBP. KBP training uses LDC2015E29, LDC2015E68 and LDC2016E64, and testing uses the KBP 2017 set LDC2017E51. The primary metric is the CoNLL score (average of MUC, B3 and CEAFe), with LEA in the appendix.

  • CoNLL scores by resolver:

    ResolverACEOntoNotesKBP
    Stanford38.665.526.2
    SpanBERT-l78.979.675.8

    Mention detection F-score rises in parallel: on OntoNotes it goes from 80.1 to 88.2.

  • Perturbations: Changes inside mentions hurt far more than replacing verbs, adjectives or adverbs in the context. The worst case is replacing names with unseen ones: OntoNotes CoNLL fell from 73.0 to 46.2 for ELMo and from 79.6 to 65.7 for SpanBERT-l.

  • Mention detection vs. coreference: Drops in coreference performance consistently track drops in mention detection.

  • Robustness: Neither ELMo nor SpanBERT-l is clearly more robust to perturbations. SpanBERT-l nevertheless always achieves higher coreference scores at similar mention detection quality, which indicates better span representations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

13 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.536

@inproceedings{lu2020conundrums,
  title = {{Conundrums in Entity Coreference Resolution: Making Sense of the State of the Art}},
  author = {Lu, Jing and Ng, Vincent},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.536},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.536}
}

References

31 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the LREC Workshop on Linguistic Coreference, pages 563–566. find our card
  2. Mohit Bansal and Dan Klein. 2012. Coreference semantics from Web features. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 389–398. find our card
  3. Anders Björkelund and Jonas Kuhn. 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 47–57. find our card
  4. Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256–2262. find our card
  5. Dennis Connolly, John D. Burger, and David S. Day. 1994. A machine learning approach to anaphoric reference. In Proceedings of International Conference on New Methods in Language Processing, pages 255–261. find
  6. Pascal Denis and Jason Baldridge. 2008. Specialized models and ranking for coreference resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 660–669. find our card
  7. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1971–1982. find our card
  8. Eraldo Fernandes, Cı́cero dos Santos, and Ruy Milidiú. 2012. Latent structure perceptron with feature induction for unrestricted coreference resolution. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 41–48. find our card
  9. Hannaneh Hajishirzi, Leila Zilles, Daniel S. Weld, and Luke Zettlemoyer. 2013. Joint coreference resolution and named-entity linking with multi-pass sieves. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 289–299. find our card
  10. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 57–60. find our card
  11. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, (8):64–77. ↗ our card
  12. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5803–5808. find our card
  13. Ben Kantor and Amir Globerson. 2019. Coreference resolution with entity equalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 673–677. find our card
  14. Andrew Kehler, Douglas Appelt, Lara Taylor, and Aleksandr Simma. 2004. The (non)utility of predicate-argument frequencies for pronoun interpretation. In Proceedings of the 2004 Human Language Technology Conference/North American Chapter of the Association for Computational Linguistics Annual Meeting, pages 289–296. find
  15. Jonathan K. Kummerfeld and Dan Klein. 2013. Error-driven analysis of challenges in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 265–277. find our card
  16. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197. find our card
  17. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692. find our card
  18. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32. find our card
  19. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics, 3:405–418. ↗ our card
  20. Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? a proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 632–642. find our card
  21. Vincent Ng. 2007. Semantic class induction and coreference resolution. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 536–543. find our card
  22. Haoruo Peng, Kai-Wei Chang, and Dan Roth. 2015. A joint framework for coreference resolution and mention head detection. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning, pages 12–21. find our card
  23. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543. find
  24. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227– 2237. find
  25. Massimo Poesio, Olga Uryupina, Renata Vieira, Mijail Alexandrov-Kabadjov, and Rodrigo Goulart. 2004. Discourse-new detectors for definite description resolution: A survey and a preliminary proposal. In Proceedings of the Conference on Reference Resolution and Its Applications, pages 47–54. find
  26. Simone Paolo Ponzetto and Michael Strube. 2007. Knowledge derived from Wikipedia for computing semantic relatedness. Journal of Artificial Intelligence Research, 30:181–212. ↗
  27. Emili Sapena, Lluı́s Padró, and Jordi Turmo. 2013. A constraint-based hypergraph partitioning approach to coreference resolution. Computational Linguistics, 39(4):847–884. ↗ our card
  28. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 656–664. find our card
  29. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the Sixth Message Understanding Conference, pages 45–52. find our card
  30. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. ACE 2005 multilingual training corpus LDC2006T06. Philadelphia: Linguistic Data Consortium. find
  31. Rui Zhang, Cı́cero Nogueira dos Santos, Michihiro Yasunaga, Bing Xiang, and Dragomir Radev. 2018. Neural coreference resolution with deep biaffine attention by joint mention detection and mention clustering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 102–107. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/lu-2020-conundrums-entity-coreference-resolution-making/