Elamur Research
☰
home / papers / 2016 / Long Short-Term Memory-Networks for Machine Reading

Long Short-Term Memory-Networks for Machine Reading

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D16-1053.pdf

Summary

Recurrent networks such as LSTMs compress the whole input into one memory vector and update their state step by step from the previous state alone. This makes long sequences hard to retain and gives no explicit way to model relations between words. Cheng, Dong and Lapata propose the Long Short-Term Memory-Network (LSTMN). It still reads text incrementally from left to right, but it replaces the single LSTM memory cell with a memory network. Each token keeps its own hidden and memory slot. At every step, an attention layer (intra-attention) weighs how much each earlier token matters for the current one. These soft, undirected relations are not supervised directly and are learned only through the downstream task. The authors also attach the LSTMN to an encoder-decoder setup through shallow and deep fusion with inter-attention. When evaluated on language modeling, sentiment classification and the task of inferring entailment between sentences, the model matches or exceeds the state of the art and beats vanilla LSTMs. On SNLI, deep fusion reaches 86.3% accuracy. The work shows that attention inside a recurrent encoder can weakly induce token-level structure.

Key points

  • The core idea is to swap the LSTM memory cell for a growing memory tape with one slot per token. This allows non-Markov state updates, with intra-attention over previous hidden states.
  • Two encoder-decoder integrations are proposed: shallow fusion and deep fusion. In deep fusion, inter-attention vectors are also memorized recurrently through a gate, and it performed slightly better.
  • For NLI, the authors used the SNLI corpus with 549,367 training pairs, 9,842 development pairs and 9,824 test pairs. The vocabulary has 36,809 words, and embeddings were initialized with 300-D GloVe 840B vectors.
  • SNLI test accuracy for LSTMN deep fusion was 86.3% (h=450, 3.4M parameters). This compares with 86.1% for mLSTM (Wang and Jiang, 2016; 1.9M parameters), 83.5% for LSTM-att and 77.6% for an LSTM encoder baseline.
  • The authors caution that their best model has substantially more parameters than competing systems, so the SNLI gain should be interpreted carefully.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

128 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 114 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1053 · arXiv 1601.06733

@inproceedings{cheng2016long,
  title = {{Long Short-Term Memory-Networks for Machine Reading}},
  author = {Cheng, Jianpeng and Dong, Li and Lapata, Mirella},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1053},
  eprint = {1601.06733},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d16-1053}
}

References

46 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural networks for question answering. In Proceedings of the 2016 NAACL: HLT, pages 1545–1554, San Diego, California. find our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In Proceedings of the 2014 ICLR, Banff, Alberta. find our card
  3. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166. find
  4. Phil Blunsom, Edward Grefenstette, and Nal Kalchbrenner. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd ACL, pages 655–665, Baltimore, Maryland. find our card
  5. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 EMNLP, pages 22–32, Lisbon, Portugal. find
  6. Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In Proceedings of the 54th ACL, pages 1466–1477, Berlin, Germany. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmi… find
  7. Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2015. Gated feedback recurrent neural networks. In Proceedings of the 32nd ICML, pages 2067–2075, Lille, France. find
  8. Peter Clark, Phil Harrison, and Niranjan Balasubramanian. 2013. A study of the knowledge base requirements for passing an elementary science test. In Proceedings of the 3rd Workshop on Automated KB Construction, San Francisco, California. find
  9. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Proceedings of the PASCAL Challenges Workshop on Recognising Textual Entailment. Sreerupa Das, C. Lee Giles, and Guo zheng Sun. 1992. Learning context-free grammars: Capabilities and limitat… find
  10. Morgan Kaufmann Publishers. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the 53rd ACL, pages 334–343, Beijing, China. find our card
  11. Oren Etzioni, Anthony Fader, Janara Christensen, Stephen Soderland, and Mausam. 2011. Open information extraction: The second generation. In Proceedings of the 22nd IJCAI, pages 3–10, Barcelona, Spain. find
  12. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of the 2011 EMNLP, pages 1535–1545, Edinburgh, Scotland, UK. find
  13. Fernanda Ferreira and John M. Henderson. 1991. Recovery from misanalyses of garden-path sentences. Journal of Memory and Language, 30:725–745. find
  14. Stefan L. Frank and Rens Bod. 2011. Insensitivity of the human sentence-processing system to hierarchical structure. Pyschological Science, 22(6):829–834. find
  15. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. find
  16. Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. 2015. Learning to transduce with unbounded memory. In Advances in Neural Information Processing Systems, pages 1819–1827. find
  17. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1684–1692. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term… find our card
  18. Sepp Hochreiter. 1991. Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Universität München. find
  19. Ozan Irsoy and Claire Cardie. 2014. Deep recursive neural networks for compositionality in language. In Advances in Neural Information Processing Systems, pages 2096–2104. find
  20. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 EMNLP, pages 1746–1751, Doha, Qatar. find our card
  21. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 2015 ICLR, San Diego, California. find
  22. Dan Klein and Christopher Manning. 2004. Corpus-based induction of syntactic structure: Models of dependency and constituency. In Proceedings of the 42nd ACL, pages 478–485, Barcelona, Spain. find
  23. Lars Konieczny. 2000. Locality and parsing complexity. Journal of Psycholinguistics, 29(6):627–645. Jan Koutnı́k, Klaus Greff, Faustino Gomez, and Jürgen Schmidhuber. 2014. A clockwork RNN. In Proceedings of the 31st ICML, pages 1863–1871, Beijing, China. find
  24. Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In Proceedings of the 33rd ICML, New York, NY. find our card
  25. Quoc V Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31st ICML, pages 1188–1196, Beijing, China. find our card
  26. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2015. Molding cnns for text: non-linear, non-consecutive convolutions. In Proceedings of the 2015 EMNLP, pages 1565–1575, Lisbon, Portugal. find
  27. Fandong Meng, Zhengdong Lu, Zhaopeng Tu, Hang Li, and Qun Liu. 2015. A deep memory-based architecture for sequence-to-sequence learning. In Proceedings of ICLR-Workshop 2016, San Juan, Puerto Rico. find
  28. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of 11th Interspeech, pages 1045–1048, Makuhari, Japan. find
  29. Tomas Mikolov, Armand Joulin, Sumit Chopra, Michael Mathieu, and Marc’Aurelio Ranzato. 2015. Learning longer memory in recurrent neural networks. In Proceedings of ICLR Workshop, San Diego, California. find
  30. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th ICML, pages 1310–1318, Atlanta, Georgia. find
  31. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 EMNLP, pages 1532–1543, Doha, Qatar. find
  32. Hoifung Poon and Pedro Domingos. 2010. Unsupervised ontology induction from text. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 296–305, Uppsala. find
  33. Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3):372–422. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In Proceedin… find
  34. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 EMNLP, pages 379–389, Lisbon, Portugal. find
  35. Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y Ng. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems, pages 801–809. find
  36. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013a. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 EMNLP, pages 1631–1642, Seattle, Washington. find
  37. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 EMNLP, pages 1631–1642, Seattle, Washingtton. find
  38. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems, pages 2431– 2439. find
  39. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd ACL, pages 1556–1566, Beijing, China. find
  40. Michael K. Tanenhaus, Michael J. Spivey-Knowlton, Kathleen M. Eberhard, and Julue C. Sedivy. 1995. Integration of visual and linguistic information in spoken language comprehension. Science, 268:1632–1634. find
  41. Ke Tran, Arianna Bisazza, and Christof Monz. 2016. Recurrent memory network for language modeling. In Proceedings of the 15th NAACL, San Diego, CA. find
  42. Shuohang Wang and Jing Jiang. 2016. Learning natural language inference with lstm. In Proceedings of the 2016 NAACL: HLT, pages 1442–1451, San Diego, California. find
  43. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. In Proceedings of the 2015 ICLR, San Diego, USA. find
  44. Caiming Xiong, Stephen Merity, and Richard Socher. 2016. Dynamic memory networks for visual and textual question answering. In Proceedings of the 33rd ICML, New York, NY. find
  45. Kaisheng Yao, Trevor Cohn, Katerina Vylomova, Kevin Duh, and Chris Dyer. 2015. Depth-gated recurrent neural networks. arXiv preprint arXiv:1508.03790. — [arXiv:1508.03790](https://arxiv.org/abs/1508.03790) find
  46. Wojciech Zaremba and Ilya Sutskever. 2014. Learning to execute. arXiv preprint arXiv:1410.4615. find