Elamur Research⌕
☰
home / papers / 2016 / Deep Reinforcement Learning for Mention-Ranking Coreference Models

Deep Reinforcement Learning for Mention-Ranking Coreference Models

Summary

Coreference systems usually make many local linking decisions, but standard evaluation metrics only judge the final clustering. As a result, training typically relies on hand-crafted losses whose error-cost hyperparameters need tuning for each language and dataset. Clark and Manning replace this tuning with reinforcement learning. They train the neural mention-ranking model of Clark and Manning (2016) to optimize a coreference metric (B³) directly. They compare two strategies: REINFORCE with a lower-variance gradient estimate, and a max-margin loss whose slack term is set by how much each candidate link would reduce the final reward. This is cheap to compute because mention-ranking decisions are independent of one another. On the English and Chinese parts of the CoNLL 2012 Shared Task, REINFORCE barely improves on the heuristic loss, while reward rescaling gives clear gains and surpasses the previous state of the art. The approach removes per-dataset cost tuning and teaches the model to avoid its most damaging mistakes. Code and models are released as deep-coref.

Key points

  • Reward-rescaled training reaches an average F1 (MUC, B³, CEAF φ4) of 65.73 on the English CoNLL 2012 test set, compared with 65.29 for Clark & Manning (2016). On Chinese it reaches 63.88.
  • The heuristic baseline (Wiseman et al., 2015) needed a grid search over error costs. The best English setting was (α_FN, α_FA, α_WL) = (0.8, 0.4, 1.0). The average reward-derived costs (0.79 for FN, 0.38 for FA) nearly match it, but costs vary widely within each error type.
  • REINFORCE gains little, likely because it optimizes expected reward but is decoded greedily at test time.
  • Reward rescaling makes roughly as many errors as the heuristic loss (e.g., 1247 vs 1258 wrong links) but less severe ones. It has fewer false-new errors on proper nouns and fewer wrong links for pronouns in telephone conversations, where the average cost is 1.21.
  • Predicted mentions come from Stanford's rule-based system. The network uses three ReLU layers (1000/500/500), RMSprop, and dropout of 0.5.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

132 papers of our library cite it (of the 140673 whose bibliographies we read); 52 of them have a page here, 80 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1245 · arXiv 1609.08667

@inproceedings{clark2016deep,
  title = {{Deep Reinforcement Learning for Mention-Ranking Coreference Models}},
  author = {Clark, Kevin and Manning, Christopher D.},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1245},
  eprint = {1609.08667},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d16-1245}
}

References

25 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The First International Conference on Language Resources and Evaluation Workshop on Linguistics Coreference, pages 563–566. find our card
  2. Anders Björkelund and Jonas Kuhn. 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Association of Computational Linguistics (ACL). find our card
  3. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Association for Computational Linguistics (ACL). find our card
  4. Kevin Clark and Christopher D. Manning. 2016. Improving coreference resolution with entity-level distributed representations. In Association for Computational Linguistics (ACL). find
  5. Hal Daumé III, John Langford, and Daniel Marcu. 2009. Search-based structured prediction. Machine Learning, 75(3):297–325. ↗
  6. Hal Daumé III. 2006. Practical structured learning techniques for natural language processing. Ph.D. thesis, University of Southern California, Los Angeles, CA. find
  7. Pascal Denis and Jason Baldridge. 2007. A ranking approach to pronoun resolution. In International Joint Conferences on Artificial Intelligence (IJCAI), pages 1588–1593. find
  8. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Empirical Methods in Natural Language Processing (EMNLP), pages 1971–1982. find our card
  9. Greg Durrett, David Leo Wright Hall, and Dan Klein. 2013. Decentralized entity-level modeling for coreference resolution. In Association for Computational Linguistics (ACL), pages 114–124. find our card
  10. Eraldo Rezende Fernandes, Cı́cero Nogueira Dos Santos, and Ruy Luiz Milidiú. 2012. Latent structure perceptron with feature induction for unrestricted coreference resolution. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Conference on Computational Natural Language Learning - Shared Task, pages 41–48. find our card
  11. Aria Haghighi and Dan Klein. 2010. Coreference resolution in a modular, entity-centered model. In Human Language Technology and North American Association for Computational Linguistics (HLT-NAACL), pages 385–393. find our card
  12. Geoffrey Hinton and Tijmen Tieleman. 2012. Lecture 6.5-RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4. find
  13. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing coadaptation of feature detectors. arXiv preprint arXiv:1207.0580. ↗
  14. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Empirical Methods in Natural Language Processing (EMNLP), pages 25–32. find our card
  15. Chao Ma, Janardhan Rao Doppa, J Walker Orr, Prashanth Mannem, Xiaoli Fern, Tom Dietterich, and Prasad Tadepalli. 2014. Prune-and-score: Learning for greedy coreference resolution. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  16. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics (TACL), 3:405–418. ↗ our card
  17. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning (ICML), pages 807–814. find
  18. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Conference on Computational Natural Language Learning - Shared Task, pages 1–40. find our card
  19. Karthik Raghunathan, Heeyoung Lee, Sudarshan Rangarajan, Nathanael Chambers, Mihai Surdeanu, Dan Jurafsky, and Christopher Manning. 2010. A multi-pass sieve for coreference resolution. In Empirical Methods in Natural Language Processing (EMNLP), pages 492–501. find our card
  20. Altaf Rahman and Vincent Ng. 2009. Supervised models for coreference resolution. In Empirical Methods in Natural Language Processing (EMNLP), pages 968– 977. find our card
  21. Richard S Sutton and Andrew G Barto. 1998. Reinforcement learning: An introduction. MIT Press. ↗
  22. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the 6th conference on Message understanding, pages 45–52. find our card
  23. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256. ↗
  24. Sam Wiseman, Alexander M Rush, Stuart M Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Association of Computational Linguistics (ACL), pages 92–100. find our card
  25. Sam Wiseman, Alexander M Rush, Stuart M Shieber, and Jason Weston. 2016. Learning global features for coreference resolution. In Human Language Technology and North American Association for Computational Linguistics (HLT-NAACL). find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/clark-2016-deep-reinforcement-learning-mention-ranking/