Elamur Research
☰
home / papers / 2015 / Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1506.07285

Summary

The authors argue that most NLP problems can be reframed as answering a question about some input text, whether the task is tagging, classification, translation or coreference. They propose one neural architecture for all of them, the Dynamic Memory Network (DMN). It has four parts:

  • an input module that encodes text with a GRU,
  • a question module that reuses the same word embeddings,
  • an episodic memory module that repeatedly scans the encoded facts using gated attention, conditioned on the question and its current memory,
  • an answer module, another GRU, that emits output once or at every word.

The system is trained end-to-end using only word vectors and input–question–answer triplets. The paper reports state-of-the-art results on Facebook's bAbI QA tasks, the Stanford Sentiment Treebank and WSJ-PTB part-of-speech tagging. Several memory passes are essential for multi-step reasoning in bAbI, help slightly for sentiment and barely matter for POS tagging. The work is an early step toward a single general model for many NLP tasks.

Key points

  • The architecture uses GRUs throughout. LSTMs gave similar accuracy at higher cost, and both clearly beat a plain tanh RNN.
  • Attention is a two-layer feed-forward gate over rich similarity features between each fact, the memory and the question. On bAbI, where supporting facts are annotated, the gates also receive supervised training.
  • The memory is initialised with the question vector. An appended end-of-passes token tells the model when to stop iterating; datasets without gate supervision use a fixed maximum number of passes.
  • On the Stanford Sentiment Treebank, two passes beat one or zero. The second pass sharpens attention, for example shifting weight from "best" to "lukewarm" in context. Three passes overfit.
  • For POS tagging on WSJ-PTB, accuracy differs by less than 0.1 across different numbers of passes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Part-of-speech tagging

Cited by

168 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 151 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1506.07285 · arXiv 1506.07285

@misc{kumar2015ask,
  title = {{Ask Me Anything: Dynamic Memory Networks for Natural Language Processing}},
  author = {Kumar, Ankit and Irsoy, Ozan and Ondruska, Peter and Iyyer, Mohit and Bradbury, James and Gulrajani, Ishaan and Zhong, Victor and Paulus, Romain and Socher, Richard},
  year = {2015},
  doi = {10.48550/arxiv.1506.07285},
  eprint = {1506.07285},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1506.07285}
}

References

42 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. Learning to Compose Neural Networks for Question Answering. arXiv preprint arXiv:1601.01705, 2016. find our card
  2. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473, 2014. find our card
  3. Bordes, A., Glorot, X., Weston, J., and Bengio, Y. Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing. AISTATS, 2012. find
  4. Bowman, S. R., Potts, C., and Manning, C. D. Recursive neural networks for learning logical semantics. CoRR, abs/1406.1827, 2014. find
  5. Chen, X. and Zitnick, C. L. Learning a recurrent visual representation for image caption generation. arXiv preprint arXiv:1411.5654, 2014. — [arXiv:1411.5654](https://arxiv.org/abs/1411.5654) find
  6. Cho, K., van Merrienboer, B., Bahdanau, D., and Bengio, Y. On the properties of neural machine translation: Encoder-decoder approaches. CoRR, abs/1409.1259, 2014a. find our card
  7. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In EMNLP, 2014b. find
  8. Chung, J., Gülçehre, Ç., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555, 2014. find
  9. Dusek, J. A. and Eichenbaum, H. The hippocampus and memory for orderly stimulusrelations. Proceedings of the National Academy of Sciences, 94(13):7109–7114, 1997. find
  10. Eichenbaum, H. and Cohen, N. J. From Conditioning to Conscious Recollection: Memory Systems of the Brain (Oxford Psychology). Oxford University Press, 1 edition, 2004. ISBN 0195178041. find
  11. Elman, J. L. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7(2-3):195–225, 1991. find
  12. Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. CoRR, abs/1410.5401, 2014. find
  13. Heckers, S., Zalesak, M., Weiss, A. P., Ditman, T., and Titone, D. Hippocampal activation during transitive inference in humans. Hippocampus, 14:153–62, 2004. find
  14. Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. Teaching machines to read and comprehend. In NIPS, 2015. find our card
  15. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 9(8):1735–1780, Nov 1997. ISSN 0899-7667. find
  16. Howard, Marc W. and Kahana, Michael J. A distributed representation of temporal context. Journal of Mathematical Psychology, 46(3):269 – 299, 2002. find
  17. Irsoy, O. and Cardie, C. Modeling compositionality with multiplicative recurrent neural networks. In ICLR, 2015. find
  18. Iyyer, M., Boyd-Graber, J., Claudino, L., Socher, R., and Daumé III, H. A neural network for factoid question answering over paragraphs. In EMNLP, 2014. find
  19. Joulin, A. and Mikolov, T. Inferring algorithmic patterns with stack-augmented recurrent nets. In NIPS, 2015. find
  20. Kaiser, L. and Sutskever, I. Neural GPUs Learn Algorithms. arXiv preprint arXiv:1511.08228, 2015. find
  21. Kalchbrenner, N., Grefenstette, E., and Blunsom, P. A convolutional neural network for modelling sentences. In ACL, 2014. find our card
  22. Karpathy, A. and Fei-Fei, L. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015. find our card
  23. Kim, Y. Convolutional neural networks for sentence classification. In EMNLP, 2014. find our card
  24. Kingma, P. and Ba, Jimmy. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. find
  25. Le, Q.V. and Mikolov., T. Distributed representations of sentences and documents. In ICML, 2014. find our card
  26. Malinowski, M. and Fritz, M. A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input. In NIPS, 2014. find
  27. Marcus, M. P., Marcinkiewicz, M. A., and Santorini, B. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2), June 1993. find our card
  28. Mikolov, T. and Zweig, G. Context dependent recurrent neural network language model. In SLT, pp. 234–239. IEEE, 2012. ISBN 978-1-4673-5125-6. find
  29. Passos, A., Kumar, V., and McCallum, A. Lexicon infused phrase embeddings for named entity resolution. In Conference on Computational Natural Language Learning. Association for Computational Linguistics, June 2014. find
  30. Pennington, J., Socher, R., and Manning, C. D. Glove: Global vectors for word representation. In EMNLP, 2014. find
  31. Socher, R., Gershman, S., Perotte, A., Sederberg, P., Blei, D., and Norman, K. A bayesian analysis of dynamics in free recall. In NIPS. 2009. find
  32. Socher, R., Huang, E. H., Pennington, J., Ng, A. Y., and Manning, C. D. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In NIPS, 2011. find
  33. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013. find
  34. Søgaard, A. Semisupervised condensed nearest neighbor for part-of-speech tagging. In ACL-HLT, 2011. find
  35. Stollenga, M. F. and J. Masci, F. Gomez, J. Schmidhuber. Deep Networks with Internal Selective Attention through Feedback Connections. In NIPS, 2014. find
  36. Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In NIPS, 2014. find
  37. Tai, K. S., Socher, R., and Manning, C. D. Improved semantic representations from tree-structured long short-term memory networks. In ACL, 2015. find
  38. Weston, J., Bordes, A., Chopra, S., and Mikolov, T. Towards ai-complete question answering: A set of prerequisite toy tasks. CoRR, abs/1502.05698, 2015a. find our card
  39. Weston, J., Chopra, S., and Bordes, A. Memory networks. In ICLR, 2015b. find
  40. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A. C., Salakhutdinov, R., Zemel, R. S., and Bengio, Y. Show, attend and tell: Neural image caption generation with visual attention. CoRR, abs/1502.03044, 2015. find our card
  41. Yang, Z., He, X., Gao, J., Deng, L., and Smola, A. Stacked attention networks for image question answering. arXiv preprint arXiv:1511.02274, 2015. find
  42. Yates, A., Banko, M., Broadhead, M., Cafarella, M. J., Etzioni, O., and Soderland, S. Textrunner: Open information extraction on the web. In HLT-NAACL (Demonstrations), 2007. find