Elamur Research⌕
☰
home / papers / 2018 / Neural Models for Reasoning over Multiple Mentions Using Coreference

Neural Models for Reasoning over Multiple Mentions Using Coreference

Summary

Answering questions about text often requires combining facts about one entity that appear far apart, possibly across sentences or documents. Standard RNN layers tend to favour nearby context, so they handle this poorly, especially with little training data. The authors propose Coref-GRU (C-GRU), a GRU variant that takes coreference clusters from an external resolver as input. Each token's update draws on two sources: the previous token's hidden state and the state of its most recent coreferent antecedent. A learned scalar weight balances the two, so a pronoun, for example, can rely mostly on its antecedent. The layer can also be seen as a memory network whose read and write positions come from coreference links rather than being learned. Replacing the bidirectional GRUs in the Gated-Attention (GA) reader with C-GRUs improves results on bAbi-1K, Wikihop and LAMBADA. The gains are largest when training data is scarce, and the model sets a new state of the art on LAMBADA. This shows that even noisy coreference annotations can supply a useful inductive bias for reasoning over multiple mentions.

Key points

  • bAbi-1K (20 tasks):
    • GA w/ C-GRU reaches 0.870 average accuracy with 5 failed tasks.
    • GA w/ GRU reaches 0.764 with 10 failed tasks.
    • EntNets reach 0.704 (max) with 15 failed tasks.
    • The largest gains are on tasks 2, 3 and 16.
  • Ablations on bAbi:
    • Adding coreference as 1-hot input features to a plain GRU does not help.
    • Removing coreference links, or replacing them with random links, causes a sharp drop in accuracy.
    • Together these show that both the structural bias and the coreference signal matter.
  • Wikihop:
    • Coreference comes from the Berkeley Entity Resolution system (about 62% F1 on CoNLL 2011).
    • C-GRU clearly helps with 1K and 5K training examples, but with the full training set the two models perform comparably.
    • Both GA models far outperform the BiDAF result reported in prior work.
  • LAMBADA:
    • Coreference comes from Stanford CoreNLP (0.63 F1).
    • GA w/ C-GRU scores 0.5569 overall and 0.6888 on the context subset.
    • GA w/ GRU scores 0.5398 and 0.6677, and Chu et al. (2017) score 0.4900 overall.
  • Cost: time complexity matches a regular GRU, while memory grows to O(BTCd) because a hidden state is kept for each coreference cluster.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

43 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 34 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-2007 · arXiv 1804.05922

@inproceedings{dhingra2018neural,
  title = {{Neural Models for Reasoning over Multiple Mentions Using Coreference}},
  author = {Dhingra, Bhuwan and Jin, Qiao and Yang, Zhilin and Cohen, William and Salakhutdinov, Ruslan},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-2007},
  eprint = {1804.05922},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n18-2007}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Huadong Chen, Shujian Huang, David Chiang, and Jiajun Chen. 2017. Improved neural machine translation with a syntax-aware encoder and decoder. ACL. find our card
  2. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. find
  3. Zewei Chu, Hai Wang, Kevin Gimpel, and David McAllester. 2017. Broad context language modeling as reading comprehension. EACL. find
  4. Kevin Clark and Christopher D Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume… find our card
  5. The official leaderboard at http://qangaroo.cs.ucl.ac.uk/leaderboard.html shows two models with better performance than reported here (as of April 2018). Since we were unable to find publications for these models we omit them here. ↗
  6. Michał Daniluk, Tim Rocktäschel, Johannes Welbl, and Sebastian Riedel. 2017. Frustratingly short attention spans in neural language modeling. ICLR. find
  7. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532– 1543. find
  8. Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. 2017a. Gated-attention readers for text comprehension. ACL. find
  9. Bhuwan Dhingra, Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. 2017b. Linguistic knowledge as memory for recurrent neural networks. arXiv preprint arXiv:1703.02620. — [arXiv:1703.02620](https://arxiv.org/abs/1703.02620) find
  10. Feng Qian, Lei Sha, Baobao Chang, Lu-chen Liu, and Ming Zhang. 2017. Syntax aware lstm model for chinese semantic role labeling. arXiv preprint arXiv:1704.00405. — [arXiv:1704.00405](https://arxiv.org/abs/1704.00405) find
  11. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In EMNLP, pages 1971–1982. find our card
  12. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017a. Bidirectional attention flow for machine comprehension. ICLR. find our card
  13. Chris Dyer. 2017. Should neural network architecture reflect linguistic structure? CoNLL Keynote. find
  14. Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. 2017b. Query-reduction networks for question answering. ICLR. find
  15. Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2016. Tracking the world state with recurrent entity networks. arXiv preprint arXiv:1612.03969. find
  16. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440–2448. find
  17. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701. find our card
  18. Swabha Swayamdipta. 2017. Learning Algorithms for Broad-Coverage Semantic Parsing. Ph.D. thesis, Carnegie Mellon University Pittsburgh, PA. find
  19. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. ACL. find
  20. Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi, and Noah A Smith. 2017. Dynamic entity representations in neural language models. EMNLP. find
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NIPS. find
  22. Sosuke Kobayashi, Ran Tian, Naoaki Okazaki, and Kentaro Inui. 2016. Dynamic entity representation with max-pooling improves machine reading. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 850–85… find
  23. Hai Wang, Takeshi Onishi, Kevin Gimpel, and David McAllester. 2017. Emergent logical structure in vector representations of neural readers. 2nd Workshop on Representation Learning for NLP, ACL. find
  24. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. EMNLP. find our card
  25. Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Fastqa: A simple and efficient neural architecture for question answering. CoNLL. find
  26. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126. find
  27. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2017. Constructing datasets for multi-hop reading comprehension across documents. arXiv preprint arXiv:1710.06481. find our card
  28. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698. find our card
  29. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The lambada dataset: Word prediction requiring a broad discourse context. ACL. find our card
  30. Sam Wiseman, Alexander M Rush, and Stuart M Shieber. 2016. Learning global features for coreference resolution. NAACL. find our card
  31. Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017. Cross-sentence n-ary relation extraction with graph lstms. Transactions of the Association for Computational Linguistics, 5:101–115. find
  32. Zichao Yang, Phil Blunsom, Chris Dyer, and Wang Ling. 2017. Reference-aware language models. EMNLP. find

↑ back to top

This page in Markdown →