Elamur Research
☰
home / papers / 2016 / Improving Coreference Resolution by Learning Entity-Level Distributed Representations

Improving Coreference Resolution by Learning Entity-Level Distributed Representations

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P16-1061.pdf

Summary

Coreference resolution means grouping the mentions in a text that refer to the same entity, so it is really a clustering task. Even so, many strong systems only score pairs of mentions and ignore information about whole entities. Clark and Manning built a neural system that learns dense vectors for pairs of coreference clusters. First, a feedforward network with three ReLU layers encodes each mention pair from word embeddings and a small set of hand-built features. A cluster-pair vector is then made by combining max-pooling and average-pooling over every mention pair that links the two clusters. The authors first train a mention-ranking model. It supplies the starting weights for the cluster ranker and removes unlikely merges in advance. The cluster ranker then merges clusters one at a time, handling the easiest decisions first. It is trained with a SEARN-style learning-to-search algorithm, so it learns how each merge affects the final partition. On the English and Chinese parts of the CoNLL 2012 Shared Task, the system clearly beats earlier state-of-the-art results. Because it needs little feature engineering, it is easy to adapt to other languages.

Key points

  • CoNLL F1 on the test sets: 65.29 for English, up from 64.21 for Wiseman et al. (2016), and 63.66 for Chinese, up from 60.06 for Björkelund & Kuhn (2014). The authors' own NN Mention Ranker scores 64.76 and 63.09.
  • Setup: 50-dimensional word2vec embeddings trained on Gigaword for English and 64-dimensional Polyglot embeddings for Chinese. Hidden layers have 1000 and 500 units. Training uses RMSProp, L2 regularization and dropout of 0.5.
  • In ablations, training only on gold (always-correct) decision sequences, without learning to search, clearly lowers scores, because the model never learns to recover from its own mistakes. Using the easy-first order also improves the cluster ranker.
  • Embeddings help with semantic links. For nominal pairs whose head words differ (e.g. "the nation" and "the country"), F1 rises to 18.9 from 10.7 for the authors' earlier statistical system.
  • Code and trained models are released as deep-coref on GitHub (clarkkev/deep-coref).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

119 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 101 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1061 · arXiv 1606.01323

@inproceedings{clark2016improving,
  title = {{Improving Coreference Resolution by Learning Entity-Level Distributed Representations}},
  author = {Clark, Kevin and Manning, Christopher D.},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1061},
  eprint = {1606.01323},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p16-1061}
}

References

38 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Empirical Methods in Natural Language Processing (EMNLP), pages 1971–1982. find our card
  2. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual NLP. Conference on Natural Language Learning (CoNLL), pages 183–192. find our card
  3. Greg Durrett, David Leo Wright Hall, and Dan Klein. 2013. Decentralized entity-level modeling for coreference resolution. In Association for Computational Linguistics (ACL), pages 114–124. find
  4. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The First International Conference on Language Resources and Evaluation Workshop on Linguistics Coreference, pages 563–566. find our card
  5. Eraldo Rezende Fernandes, Cı́cero Nogueira Dos Santos, and Ruy Luiz Milidiú. 2012. Latent structure perceptron with feature induction for unrestricted coreference resolution. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Conference on Computational Natural Language Learning - Shared Task, pages 41–48. find our card
  6. Eric Bengtson and Dan Roth. 2008. Understanding the value of features for coreference resolution. In Empirical Methods in Natural Language Processing (EMNLP), pages 294–303. find our card
  7. Anders Björkelund and Jonas Kuhn. 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Association of Computational Linguistics (ACL), pages 47–57. find
  8. Aria Haghighi and Dan Klein. 2010. Coreference resolution in a modular, entity-centered model. In Human Language Technology and North American Association for Computational Linguistics (HLT-NAACL), pages 385–393. find
  9. Kai-Wei Chang, He He, Hal Daumé III, and John Langford. 2015a. Learning to search for dependencies. arXiv preprint arXiv:1503.05615. ↗
  10. Geoffrey Hinton and Tijmen Tieleman. 2012. Lecture 6.5-RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4. find
  11. Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé III, and John Langford. 2015b. Learning to search better than your teacher. In International Conference on Machine Learning (ICML). find
  12. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing coadaptation of feature detectors. arXiv preprint arXiv:1207.0580. ↗
  13. Chen Chen and Vincent Ng. 2012. Combining the best of two worlds: A hybrid approach to multilingual coreference resolution. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Conference on Computational Natural Language Learning - Shared Task, pages 56–63. find
  14. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. ↗
  15. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the Bell tree. In Association for Computational Linguistics (ACL), page 135. find
  16. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Association for Computational Linguistics (ACL). find our card
  17. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Empirical Methods in Natural Language Processing (EMNLP), pages 25–32. find our card
  18. Hal Daumé III and Daniel Marcu. 2005. A large-scale exploration of effective global features for a joint entity detection and tracking model. In Empirical Methods in Natural Language Processing (EMNLP), pages 97–104. find
  19. Chao Ma, Janardhan Rao Doppa, J Walker Orr, Prashanth Mannem, Xiaoli Fern, Tom Dietterich, and Prasad Tadepalli. 2014. Prune-and-score: Learning for greedy coreference resolution. In Empirical Methods in Natural Language Processing (EMNLP). find
  20. Hal Daumé III, John Langford, and Daniel Marcu. 2009. Search-based structured prediction. Machine Learning, 75(3):297–325. find
  21. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics (TACL), 3:405–418. find our card
  22. Hal Daumé III, John Langford, and Stephane Ross. 2014. Efficient programmable learning to search. arXiv preprint arXiv:1406.1837. ↗
  23. Andrew McCallum and Ben Wellner. 2003. Toward conditional models of identity uncertainty with application to proper noun coreference. In Proceedings of the IJCAI Workshop on Information Integration on the Web. find
  24. Pascal Denis and Jason Baldridge. 2007. A ranking approach to pronoun resolution. In International Joint Conferences on Artificial Intelligence (IJCAI), pages 1588–1593. antecedent ranking features for coreference resolution. In Association of Computational Linguistics (ACL), pages 92–100. find
  25. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (NIPS), pages 3111–3119. find
  26. Sam Wiseman, Alexander M Rush, Stuart M Shieber, and Jason Weston. 2016. Learning global features for coreference resolution. In Human Language Technology and North American Association for Computational Linguistics (HLT-NAACL). find our card
  27. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning (ICML), pages 807–814. find
  28. Xiaofeng Yang, Jian Su, Jun Lang, Chew Lim Tan, Ting Liu, and Sheng Li. 2008. An entity-mention model for coreference resolution with inductive logic programming. In Association of Computational Linguistics (ACL), pages 843–851. find
  29. Vincent Ng and Claire Cardie. 2002. Improving machine learning approaches to coreference resolution. In Association of Computational Linguistics (ACL), pages 104–111. find our card
  30. Haoruo Peng, Kai-Wei Chang, and Dan Roth. 2015. A joint framework for coreference resolution and mention head detection. Conference on Natural Language Learning (CoNLL), 51:12. find
  31. Hoifung Poon and Pedro Domingos. 2008. Joint unsupervised coreference resolution with markov logic. In Empirical Methods in Natural Language Processing (EMNLP), pages 650–659. find
  32. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Conference on Computational Natural Language Learning - Shared Task, pages 1–40. find our card
  33. Karthik Raghunathan, Heeyoung Lee, Sudarshan Rangarajan, Nathanael Chambers, Mihai Surdeanu, Dan Jurafsky, and Christopher Manning. 2010. A multi-pass sieve for coreference resolution. In Empirical Methods in Natural Language Processing (EMNLP), pages 492–501. find our card
  34. Altaf Rahman and Vincent Ng. 2011. Narrowing the modeling gap: a cluster-ranking approach to coreference resolution. Journal of Artificial Intelligence Research (JAIR), pages 469–521. find
  35. Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong Lim. 2001. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics, 27(4):521–544. find our card
  36. Veselin Stoyanov and Jason Eisner. 2012. Easy-first coreference resolution. In International Conference on Computational Linguistics (COLING), pages 2519–2534. find
  37. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the 6th conference on Message understanding, pages 45–52. find our card
  38. Sam Wiseman, Alexander M Rush, Stuart M Shieber, and Jason Weston. 2015. Learning anaphoricity and find