Elamur Research⌕
☰
home / papers / 2018 / Neural Coreference Resolution with Deep Biaffine Attention by Joint Mention Detection and Mention Clustering

Neural Coreference Resolution with Deep Biaffine Attention by Joint Mention Detection and Mention Clustering

Summary

This short paper extends the end-to-end neural coreference resolver of Lee et al. (2017). That system treats every text span as a candidate mention and learns to choose an antecedent for each one. The authors identify two weaknesses. First, antecedent scores come from plain feed-forward networks. Second, training uses only the marginal likelihood of gold clusters, so mention detection gets only indirect supervision.

They propose two changes:

  • Biaffine attention scorer (after Dozat and Manning, 2017). It separately models how compatible two spans are and how likely a span is to have any antecedent at all.
  • Joint training objective. An explicit binary mention-detection loss is added to the clustering log-likelihood, so the model learns directly which spans are real entity mentions.

Span representations are the same as in the baseline: BiLSTMs plus head-finding attention.

When evaluated on the English test data from the CoNLL-2012 Shared Task (drawn from OntoNotes), the model achieves a new state of the art. Most of the gain comes from higher precision. The model also finds many correct mentions that never appear in the training data. This suggests it generalizes rather than memorizes, which matters for adapting coreference systems to new domains.

Key points

  • Data and metrics: CoNLL-2012 English contains 2,802 training, 343 development and 348 test documents. Evaluation uses MUC, B³ and CEAFφ4, with their average F1 serving as the CoNLL score.
  • Main results: The single model reaches 67.8 average F1, against 67.2 for Lee et al. (2017). A 5-model ensemble reaches 69.2, against 68.8. The gains are significant at p < 0.05 under a paired bootstrap test.
  • Ablation (dev set): Removing the detection loss lowers the score to 67.5. Removing biaffine attention lowers it to 67.4. The baseline scores 67.3.
  • Setup:
    • Hyperparameters follow Lee et al. (2017).
    • Spans are at most 10 words, with up to 250 antecedents considered per span.
    • Embeddings are fixed GloVe (300-d) concatenated with Turian (50-d).
    • The detection-loss weight is λ_detect = 0.1.
    • Training converges in about 200K updates, faster than the baseline.
  • Mention detection: The model correctly detects 1,048 mentions that the baseline misses. Of these, 662 do not occur in the training data. The advantage is largest for spans of 5 or more words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution · Dependency parsing

Cited by

25 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 14 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-2017 · arXiv 1805.04893

@inproceedings{zhang2018neural,
  title = {{Neural Coreference Resolution with Deep Biaffine Attention by Joint Mention Detection and Mention Clustering}},
  author = {Zhang, Rui and Santos, Cícero Nogueira dos and Yasunaga, Michihiro and Xiang, Bing and Radev, Dragomir},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-2017},
  eprint = {1805.04893},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p18-2017}
}

References

35 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The first international conference on language resources and evaluation workshop on linguistics coreference. find our card
  2. Eric Bengtson and Dan Roth. 2008. Understanding the value of features for coreference resolution. In EMNLP. find our card
  3. Anders Björkelund and Jonas Kuhn. 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In ACL. find our card
  4. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In ACL. find our card
  5. Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In EMNLP. find our card
  6. Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In ACL. find our card
  7. Hal Daumé III and Daniel Marcu. 2005. A large-scale exploration of effective global features for a joint entity detection and tracking model. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing. find
  8. Pascal Denis and Jason Baldridge. 2007a. Joint determination of anaphoricity and coreference resolution using integer programming. In NAACL. find our card
  9. Pascal Denis and Jason Baldridge. 2007b. A ranking approach to pronoun resolution. In IJCAI. find
  10. Vincent Ng. 2010. Supervised noun phrase coreference research: The first fifteen years. In ACL. find our card
  11. Timothy Dozat and Christopher D Manning. 2017. Deep biaffine attention for neural dependency parsing. In ICLR. find our card
  12. Vincent Ng and Claire Cardie. 2002. Improving machine learning approaches to coreference resolution. In ACL. find our card
  13. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In EMNLP. find our card
  14. Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. Transactions of the Association for Computational Linguistics, 2:477–490. find our card
  15. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. find
  16. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL-Shared Task. find our card
  17. Eraldo Rezende Fernandes, Cı́cero Nogueira Dos Santos, and Ruy Luiz Milidiú. 2012. Latent structure perceptron with feature induction for unrestricted coreference resolution. In Joint Conference on EMNLP and CoNLL-Shared Task. find our card
  18. Cı́cero Nogueira dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proceedings of the 31st International Conference on Machine Learning (ICML), JMLR: W&CP volume 32. find
  19. Eraldo Rezende Fernandes, Cı́cero Nogueira dos Santos, and Ruy Luiz Milidiú. 2014. Latent trees for coreference resolution. Computational Linguistics. find our card
  20. Aria Haghighi and Dan Klein. 2009. Simple coreference resolution with rich syntactic and semantic features. In EMNLP. find our card
  21. Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong Lim. 2001. A machine learning approach to coreference resolution of noun phrases. Computational linguistics, 27(4):521–544. find our card
  22. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  23. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In ACL. find our card
  24. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: the 90% solution. In NAACL. find
  25. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the 6th conference on Message understanding. find our card
  26. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  27. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In EMNLP. find
  28. Sam Wiseman, Alexander M Rush, and Stuart M Shieber. 2016. Learning global features for coreference resolution. In NAACL. find our card
  29. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the conll-2011 shared task. In Proceedings of the fifteenth conference on computational natural language learning: Shared task, pages 28–… find our card
  30. Sam Joshua Wiseman, Alexander Matthew Rush, Stuart Merrill Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In ACL. find our card
  31. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In EMNLP. find our card
  32. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In EMNLP. find our card
  33. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics, 3:405–418. find our card
  34. Nafise Sadat Moosavi and Michael Strube. 2017. Lexical features in coreference resolution: To be used with caution. In ACL. find our card
  35. Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In ICML. find

↑ back to top

This page in Markdown →