Elamur Research⌕
☰
home / papers / 2017 / Dynamic Entity Representations in Neural Language Models

Dynamic Entity Representations in Neural Language Models

Paper · 2017 · Yangfeng Ji, Chenhao Tan, Sebastian Martschat et al.

Summary

Most neural language models do not represent entities beyond the words used to mention them, so they cannot follow who or what a long text is about. The authors propose ENTITYNLM, an LSTM-based generative language model with extra latent variables. At each token, these variables decide whether the token belongs to an entity mention, which entity is referred to (a known one or a new one), and how many words remain in the mention. Each entity has its own vector. The vector is created at first mention and updated through a gated interpolation with the LSTM state each time the entity reappears. The current entity vector also shapes the next words. The model is trained on text annotated with mentions and coreference and supports several uses. It beats strong n-gram and RNN language models on the CoNLL 2012 English test set and improves a coreference system by reranking its k-best outputs. It also achieves state-of-the-art entity prediction on InScript. The results show that an explicit, evolving entity memory helps both text generation and discourse-level tasks.

Key points

  • Each word gets three variables: R_t (mention or not), E_t (entity index, growing to allow any number of entities), and L_t (remaining mention length, capped at 25 tokens).
  • New entity vectors are drawn from a normal distribution centered on a learned "entity" embedding r_1 (σ = 0.01) and projected onto the unit ball. Only the selected entity is updated at each step, unlike the EntNet of Henaff et al. (2016).
  • Entity choice combines the LSTM context with distance features to earlier mentions. Words are generated with a class-factorized softmax conditioned on the hidden state and the current entity vector.
  • On InScript entity prediction, ENTITYNLM reaches 74.23% accuracy. This compares with 62.65% for Modi et al. (2017), 31.08% and 45.34% for two simple baselines, and 77.35% for humans (p < 0.01).
  • Reranking k-best coreference outputs on the CoNLL 2012 test set gives results significantly better than the base system's original top choices.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States263 works · 180,581 citations🎓Heidelberg UniversityHeidelberg · Germany27 works · 40,944 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1 Language models & pretraining · 1,343 works

Cite

Cite as Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi and Noah A. Smith. 2017. Dynamic Entity Representations in Neural Language Models. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1195 · arXiv 1708.00781

@inproceedings{ji2017dynamic,
  title = {{Dynamic Entity Representations in Neural Language Models}},
  author = {Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi and Noah A. Smith},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1195},
  eprint = {1708.00781},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d17-1195}
}

References

36 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In LREC Workshop on Linguistic Coreference. our card
  2. Paul Baltescu and Phil Blunsom. 2015. Pragmatic neural language modelling in machine translation. In NAACL. find
  3. Anders Björkelund and Jonas Kuhn. 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In ACL. our card
  4. Nathanael Chambers and Daniel Jurafsky. 2008. Unsupervised Learning of Narrative Event Chains. In ACL. find
  5. Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In EMNLP. our card
  6. Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In ACL. our card
  7. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159. find
  8. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In EMNLP. find
  9. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pages 249–256. find
  10. Joshua Goodman. 2001. Classes for fast maximum entropy training. In ICASSP. find
  11. Aria Haghighi and Dan Klein. 2010. Coreference resolution in a modular, entity-centered model. In NAACL. our card
  12. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In ACL. our card
  13. Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2016. Tracking the world state with recurrent entity networks. arXiv:1612.03969. find
  14. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  15. Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer, and Jacob Eisenstein. 2016a. Document context language models. In ICLR (workshop track). find
  16. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016b. A latent variable recurrent neural network for discourse-driven language models. In NAACL-HLT. find
  17. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980. find
  18. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In HLT-EMNLP. our card
  19. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the Bell tree. In ACL. our card
  20. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics, 3:405–418. our card
  21. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH. find
  22. Ashutosh Modi, Ivan Titov, Vera Demberg, Asad Sayeed, and Manfred Pinkal. 2017. Modeling semantic expectation: Using script knowledge for referent prediction. Transactions of the Association of Computational Linguistics, 5:31–44. find
  23. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In EMNLP-CoNLL. our card
  24. Libin Shen, Anoop Sarkar, and Franz Josef Och. 2004. Discriminative reranking for machine translation. In NAACL. find
  25. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958. find
  26. Ke Tran, Arianna Bisazza, and Christof Monz. 2016. Recurrent memory networks for language modeling. In NAACL-HLT. find
  27. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In MUC. our card
  28. Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. Learning global features for coreference resolution. In NAACL. our card
  29. Zichao Yang, Phil Blunsom, Chris Dyer, and Wang Ling. 2016. Reference-aware language models. arXiv:1611.01628. find
  30. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2015. Recurrent neural network regularization. ICLR. our card
  31. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. In NAACL. find
  32. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. Dynet: The dynamic neural network toolkit. arXiv:1701.03980. our card
  33. Vincent Ng. 2005. Machine learning for coreference resolution: From local classification to global ranking. In ACL. our card
  34. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. our card
  35. Jim Pitman. 1995. Exchangeable and partially exchangeable random partitions. Probability Theory and Related Fields, 102(2):145–158. find
  36. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In ACL. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ji-2017-dynamic-entity-representations-neural-language/