Elamur Research⌕
☰
home / papers / 2016 / Gated-Attention Readers for Text Comprehension

Gated-Attention Readers for Text Comprehension

Paper · 2016 · Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang et al.

Summary

The paper addresses cloze-style reading comprehension: given a document and a query with one phrase blanked out, a model must choose the correct filler from candidate entities found in the document. The authors propose the Gated-Attention (GA) Reader, which combines two ideas that earlier work had developed separately: multi-hop reading and attention. The model stacks several layers of bidirectional GRUs over the document. At each layer, every document token attends over the query to build its own query summary, and that summary is multiplied element-wise with the token's hidden state. This acts as a filter that suppresses query-irrelevant features before the next pass. The final layer uses an attention-sum step, as in the AS Reader, to score candidates. Experiments on CNN, Daily Mail and Who Did What (WDW) show state-of-the-art accuracy, with gains above 4% over strong baselines. Ablations and statistical tests confirm that the gating itself drives much of the improvement. This makes the mechanism a candidate for other tasks where several information sources interact.

Key points

  • The GA module computes a token-specific query vector by softmax attention, then gates the document token by element-wise multiplication. On WDW, multiplication clearly beat addition and concatenation as the combining operator.
  • With one hop (K = 1), the model reduces to the AS Reader without gating. Accuracy climbs steadily up to K = 3 and then levels off.
  • In the WDW ablation (val/test), the full GA scored 68.3/68.0. Removing character embeddings dropped it to 66.9/66.9, and removing token-level query attention dropped it to 65.7/65.0. Replacing GloVe vectors (trained on about 6B tokens) with corpus-trained embeddings gave the largest drop, to 64.0/62.5.
  • The benefit of gating was assessed across 50%, 75% and 100% training-set sizes using an exact one-sided McNemar test. The gains were mostly significant (p < 0.01) on CNN, especially with more data.
  • Attention visualizations suggest that different layers focus on different clue words in the query. Code was released at github.com/bdhingra/ga-reader.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,226 citations🏢MicrosoftRedmond · United States378 works · 161,630 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

Cite

Cite as Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William W. Cohen and Ruslan Salakhutdinov. 2016. Gated-Attention Readers for Text Comprehension. arXiv:1606.01549.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1606.01549 · arXiv 1606.01549

@misc{dhingra2016gated,
  title = {{Gated-Attention Readers for Text Comprehension}},
  author = {Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William W. Cohen and Ruslan Salakhutdinov},
  year = {2016},
  doi = {10.48550/arxiv.1606.01549},
  eprint = {1606.01549},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1606.01549}
}

References

32 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
  2. Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. Embracing data abundance: Booktest dataset for reading comprehension. arXiv preprint arXiv:1610.00956, 2016. — [arXiv:1610.00956](https://arxiv.org/abs/1610.00956) find
  3. Danqi Chen, Jason Bolton, and Christopher D Manning. A thorough examination of the cnn/daily mail reading comprehension task. ACL, 2016.
  4. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. ACL, 2015. — Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
  5. Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. Attention-over-attention neural networks for reading comprehension. ACL, 2017. find
  6. Bhuwan Dhingra, Zhong Zhou, Dylan Fitzpatrick, Michael Muehl, and William W Cohen. Tweet2vec: Character-based distributed representations for social media. ACL, 2016. find
  7. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pp. 1684–1692, 2015.
  8. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. The goldilocks principle: Reading children’s books with explicit memory representations. ICLR, 2016. — The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
  9. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735– 1780, 1997. find
  10. Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. Text understanding with the attention sum reader network. ACL, 2016.
  11. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
  12. Ryan Kiros, Richard Zemel, and Ruslan R Salakhutdinov. A multiplicative model for learning distributed text-based attribute representations. In Advances in Neural Information Processing Systems, pp. 2348– 2356, 2014. find
  13. Sosuke Kobayashi, Ran Tian, Naoaki Okazaki, and Kentaro Inui. Dynamic entity representations with max-pooling improves machine reading. In NAACL-HLT, 2016. find
  14. Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. ICML, 2016.
  15. Peng Li, Wei Li, Zhengyan He, Xuguang Wang, Ying Cao, Jie Zhou, and Wei Xu. Dataset and neural recurrent sequence labeling model for open-domain factoid question answering. arXiv preprint arXiv:1607.06275, 2016. — [arXiv:1607.06275](https://arxiv.org/abs/1607.06275) find
  16. Jeff Mitchell and Mirella Lapata. Vector-based models of semantic composition. In ACL, pp. 236–244, 2008. find
  17. Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, pp. 2204– 2212, 2014. find
  18. Tsendsuren Munkhdalai and Hong Yu. Neural semantic encoders. EACL, 2017a. find
  19. Tsendsuren Munkhdalai and Hong Yu. Reasoning with memory augmented neural networks for language comprehension. ICLR, 2017b. find
  20. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. Who did what: A large-scale person-centered cloze dataset. EMNLP, 2016.
  21. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. ICML (3), 28:1310–1318, 2013. find
  22. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532– 1543, 2014. URL http://www.aclweb.org/anthology/D14-1162. ↗
  23. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. ICLR, 2017.
  24. Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. Reasonet: Learning to stop reading in machine comprehension. arXiv preprint arXiv:1609.05284, 2016. — [arXiv:1609.05284](https://arxiv.org/abs/1609.05284) find
  25. Alessandro Sordoni, Phillip Bachman, and Yoshua Bengio. Iterative alternating neural attention for machine reading. arXiv preprint arXiv:1606.02245, 2016. — [arXiv:1606.02245](https://arxiv.org/abs/1606.02245) find
  26. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In Advances in Neural Information Processing Systems, pp. 2431–2439, 2015. find
  27. Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints, abs/1605.02688, May 2016. URL http://arxiv.org/abs/1605.02688. ↗
  28. Adam Trischler, Zheng Ye, Xingdi Yuan, and Kaheer Suleman. Natural language comprehension with the epireader. EMNLP, 2016. find
  29. Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. ICLR, 2015. find
  30. Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, and Ruslan Salakhutdinov. On multiplicative integration with recurrent neural networks. Advances in Neural Information Processing Systems, 2016. find
  31. Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Learning multi-relational semantics using neural-embedding models. NIPS Workshop on Learning Semantics, 2014. find
  32. Zhilin Yang, Ruslan Salakhutdinov, and William Cohen. Multi-task cross-lingual sequence tagging from scratch. arXiv preprint arXiv:1603.06270, 2016. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dhingra-2016-gated-attention-readers-text-comprehension/