Elamur Research⌕
☰
home / papers / 2019 / End-to-end Deep Reinforcement Learning Based Coreference Resolution

End-to-end Deep Reinforcement Learning Based Coreference Resolution

Summary

Neural coreference systems that detect and cluster mentions jointly have pushed the state of the art. However, they still learn from heuristic losses over individual pairwise links. The standard evaluation metrics can only be computed once every link in a document is decided. Fei et al. (Baidu Research) address this mismatch by recasting the higher-order mention-ranking model of Lee et al. (2018) as a policy network trained with policy gradient (REINFORCE with a baseline).

Each mention's antecedent choice is an action. After a whole document is processed, the reward is the average of MUC, B³ and CEAFφ4. Training proceeds in three steps:

  • The network is first pretrained with a supervised loss.
  • It is then fine-tuned on sampled trajectories.
  • An entropy regularization term keeps the policy exploring and prevents premature convergence.

On English OntoNotes v5.0 the model reaches 73.8 average F1 with ELMo, a statistically significant gain over 73.0 for Lee et al. (2018). It is the first fully end-to-end RL coreference model, and it shows that the evaluation metrics can be optimized directly.

Key points

  • Data: English OntoNotes v5.0 (2,802 train / 343 dev / 348 test documents).
  • Main results:
    • The full model (ELMo + entropy regularization) scores 73.8 average F1, against 73.0 for Lee et al. (2018), with p < 0.05.
    • Without ELMo it reaches 70.5, compared with 69.5 for the base reinforced model.
  • Training settings:
    • About 200K pretraining steps.
    • 100 sampled trajectories per update.
    • Entropy weight λ_expr = 10⁻⁴, tuned on the dev set.
  • Dev-set ablation (full model 74.1):
    • Dropping ELMo gives 71.0, the largest loss.
    • Dropping entropy regularization gives 73.1.
    • Dropping coarse-to-fine pruning gives 73.8.
    • Dropping second-order inference gives 73.6.
  • Mention detection: F1 is 85.7 versus 84.9, driven mainly by higher precision (89.6 vs 86.2), meaning fewer false-positive links and mentions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

14 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 10 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1064

@inproceedings{fei2019end,
  title = {{End-to-end Deep Reinforcement Learning Based Coreference Resolution}},
  author = {Fei, Hongliang and Li, Xu and Li, Dingcheng and Li, Ping},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1064},
  url = {https://doi.org/10.18653/v1/p19-1064}
}

References

19 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3219–3232, Brusse… find our card
  2. Jie Cai and Michael Strube. 2010. Evaluation metrics for end-to-end coreference resolution systems. In Proceedings the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIG-DIAL), pages 28–36, Tokyo, Japan. find our card
  3. Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2256–2262, Austin, TX. find our card
  4. Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33nd International Conference on Machine Learning (ICML), New York, NY. find
  5. Kevin Clark and Christopher D Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 643–653, Berlin, Germany. find our card
  6. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  7. Zeyu Dai, Hongliang Fei, and Ping Li. 2019. Coreference aware representation learning for neural named entity recognition. In Proceedings of the International Joint Conference on Artificial Intelligence (IJ-CAI), Macau. find
  8. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–27, Portland, OR… find our card
  9. Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. 2019. Diversity is all you need: Learning skills without a reward function. In Seventh International Conference on Learning Representations (ICLR), New Orleans, LA. find
  10. Marta Recasens and Eduard Hovy. 2011. Blanc: Implementing the rand index for coreference evaluation. Natural Language Engineering, 17(4):485– 510. find
  11. Ralph Grishman and Beth Sundheim. 1995. Design of the muc-6 evaluation. In Proceedings of the 6th conference on Message understanding (MUC), pages 1– 11, Columbia, MD. find
  12. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. CoRR, abs/1502.05698. find our card
  13. Aria Haghighi and Dan Klein. 2010. Coreference resolution in a modular, entity-centered model. In Proceedings of Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics (NAACL), pages 385–393, Los Angeles, CA. find our card
  14. Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. Learning global features for coreference resolution. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 994– 1004, San Diego,… find our card
  15. Hannaneh Hajishirzi, Leila Zilles, Daniel S. Weld, and Luke S. Zettlemoyer. 2013. Joint coreference resolution and named-entity linking with multi-pass sieves. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 289–299, Seattle, WA. find our card
  16. Qingyu Yin, Yu Zhang, Weinan Zhang, Ting Liu, and William Yang Wang. 2018. Deep reinforcement learning for chinese zero pronoun resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 569–578, Melbourne, Australia. find our card
  17. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 188–197, Copenhagen, Denmark. find our card
  18. Rui Zhang, Cı́cero Nogueira dos Santos, Michihiro Yasunaga, Bing Xiang, and Dragomir R. Radev. 2018. Neural coreference resolution with deep biaffine attention by joint mention detection and mention clustering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistic… find our card
  19. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 687–692, New Orle… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/fei-2019-end-end-deep-reinforcement-learning/