Elamur Research
☰
home / papers / 2016 / Learning Global Features for Coreference Resolution

Learning Global Features for Coreference Resolution

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N16-1114.pdf

Summary

This NAACL-HLT 2016 paper tackles a long-standing puzzle in coreference resolution. Information about whole entity clusters should help, yet purely local mention-ranking systems still match or beat structured models. The authors argue that hand-designed cluster-level features are the bottleneck, since such features usually end up either overly coarse or overly sparse.

Their solution is to let a recurrent neural network learn each cluster's representation. The RNN reads, in document order, the embedded mentions already assigned to a partial cluster, and its hidden state serves as a global feature. That feature feeds an extra scoring term added to a standard mention-ranking model.

The whole system is trained end-to-end as a history-based local classifier and decodes with simple greedy left-to-right search. On the CoNLL 2012 English data it beats the prior state of the art by more than 0.8 CoNLL points, with significant gains on all three CoNLL metrics.

The work shows that global context can be exploited without manual cluster features or expensive inference. It particularly helps with pronouns, a persistent source of errors.

Key points

  • Motivation from prior error analysis: Wiseman et al. (2015) found that pronouns account for roughly 59% of mention-ranking precision errors and about 24% of recall errors on the CoNLL 2012 English dev set.
  • Non-local patterns in the training data:
    • Over 70% of pleonastic "it" and over 74% of pleonastic "you" follow an earlier pleonastic instance of the same word.
    • Over 78% of referential "I" corefer with a previous "I".
  • Architecture:
    • Mention features are embedded through a tanh layer.
    • An RNN then runs over each cluster's mentions.
    • The learned global term is combined with the local mention-ranking score.
    • No additional search is needed beyond greedy decoding.
  • Comparison with an averaging baseline: The RNN clearly outperforms "Avg", which averages mention embeddings instead of modelling them as a sequence and barely improves on plain mention ranking even with oracle history. The largest gains come on non-anaphoric pronouns and pronominal wrong-link errors.
  • What the representations capture: T-SNE plots of learned cluster states group entities by type (nominal, proper, pronominal) and by number, and also capture subtler groupings such as geographic terms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

92 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 75 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1114 · arXiv 1604.03035

@inproceedings{wiseman2016learning,
  title = {{Learning Global Features for Coreference Resolution}},
  author = {Wiseman, Sam and Rush, Alexander M. and Shieber, Stuart M.},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1114},
  eprint = {1604.03035},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n16-1114}
}

References

41 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, pages 1171– 1179. find
  2. Eric Bengtson and Dan Roth. 2008. Understanding the Value of Features for Coreference Resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 294–303. Association for Computational Linguistics. find our card
  3. Anders Björkelund and Jonas Kuhn. 2014. Learning structured perceptrons for coreference Resolution with Latent Antecedents and Non-local Features. ACL, Baltimore, MD, USA, June. find
  4. Jie Cai and Michael Strube. 2010. End-to-end coreference resolution via hypergraph partitioning. In 23rd International Conference on Computational Linguistics (COLING), pages 143–151. find
  5. Kai-Wei Chang, Rajhans Samdani, and Dan Roth. 2013. A Constrained Latent Variable Model for Coreference Resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 601–612. find
  6. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 1405–1415. find our card
  7. Aron Culotta, Michael Wick, Robert Hall, and Andrew McCallum. 2007. First-order Probabilistic Models for Coreference Resolution. In Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics (NAACL HLT). find
  8. Hal Daumé III, John Langford, and Daniel Marcu. 2009. Search-based structured prediction. Machine Learning, 75(3):297–325. find
  9. Pascal Denis and Jason Baldridge. 2008. Specialized Models and Ranking for Coreference Resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 660– 669. Association for Computational Linguistics. find
  10. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. The Journal of Machine Learning Research, 12:2121–2159. find
  11. Greg Durrett and Dan Klein. 2013. Easy Victories and Uphill Battles in Coreference Resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1971–1982. find our card
  12. Greg Durrett and Dan Klein. 2014. A Joint Model for Entity Analysis: Coreference, Typing, and Linking. Transactions of the Association for Computational Linguistics, 2:477–490. find
  13. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 334–343. find our card
  14. Eraldo Rezende Fernandes, Cı́cero Nogueira Dos Santos, and Ruy Luiz Milidiú. 2012. Latent Structure Perceptron with Feature Induction for Unrestricted Coreference Resolution. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 41–48. Association for Computational Linguistics. find our card
  15. Aria Haghighi and Dan Klein. 2010. Coreference Resolution in a Modular, Entity-centered Model. In The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 385–393. Association for Computational Linguistics. find
  16. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9:1735–1780. find
  17. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: the 90% Solution. In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers, pages 57–60. Association for Computational Linguistics. find
  18. Philipp Koehn. 2004. Statistical Significance Tests for Machine Translation Evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395. Citeseer. find
  19. Jonathan K. Kummerfeld and Dan Klein. 2013. Error-driven Analysis of Challenges in Coreference Resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, WA, USA, October. Nicholas Léonard, Yand Waghmare, Sagar ad Wang, and Jin-Hwa Kim. 2015. rnn: Recurrent Library for Torch. arXiv preprint arXiv:1511.07889. ↗ our card
  20. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in nlp. In NAACL HLT. find our card
  21. Xiaoqiang Luo, Sameer Pradhan, Marta Recasens, and Eduard Hovy. 2014. An Extension of BLANC to System Mentions. Proceedings of ACL, Baltimore, Maryland, June. find
  22. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. TACL, 3:405– 418. find our card
  23. Sebastian Martschat, Thierry Göckel, and Michael Strube. 2015. Analyzing and visualizing coreference resolution errors. In NAACL HLT, pages 6–10. find
  24. Andrew McCallum and Ben Wellner. 2003. Toward Conditional Models of Identity Uncertainty with Application to Proper Noun Coreference. Advances in Neural Information Processing Systems 17. find
  25. Vincent Ng and Claire Cardie. 2002. Identifying Anaphoric and Non-anaphoric Noun Phrases to Improve Coreference Resolution. In Proceedings of the 19th international conference on Computational linguistics-Volume 1, pages 1–7. Association for Computational Linguistics. find
  26. Haoruo Peng, Kai-Wei Chang, and Dan Roth. 2015. A joint framework for coreference resolution and mention head detection. In Proceedings of the 19th Conference on Computational Natural Language Learning (CoNLL), pages 12–21. find
  27. Hoifung Poon and Pedro M. Domingos. 2008. Joint unsupervised coreference resolution with markov logic. In 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 650–659. find
  28. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 1–40. Association for Computational Linguistics. find our card
  29. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation. In Proceedings of the Association for Computational Linguistics. find our card
  30. Altaf Rahman and Vincent Ng. 2009. Supervised Models for Coreference Resolution. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2-Volume 2, pages 968–977. Association for Computational Linguistics. find
  31. Altaf Rahman and Vincent Ng. 2011. Narrowing the modeling gap: A cluster-ranking approach to coreference resolution. J. Artif. Intell. Res. (JAIR), 40:469– 521. find
  32. Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 627–635. find
  33. Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong Lim. 2001. A Machine Learning Approach to Coreference Resolution of Noun Phrases. Computational Linguistics, 27(4):521–544. find our card
  34. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. find
  35. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  36. Veselin Stoyanov and Jason Eisner. 2012. Easy-first Coreference Resolution. In COLING, pages 2519– 2534. Citeseer. find
  37. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS), pages 3104–3112. find
  38. Laurens van der Maaten and Geoffrey E. Hinton. 2012. Visualizing non-metric similarities in multiple maps. Machine Learning, 87(1):33–55. find
  39. Sam Wiseman, Alexander M. Rush, Stuart M. Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 1416–1426. find our card
  40. Chun-Nam John Yu and Thorsten Joachims. 2009. Learning Structural SVMs with Latent Variables. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 1169–1176. ACM. find
  41. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. CoRR, abs/1409.2329. ↗ our card