Elamur Research⌕
☰
home / papers / 2017 / Gated Self-Matching Networks for Reading Comprehension and Question Answering

Gated Self-Matching Networks for Reading Comprehension and Question Answering

Summary

The paper addresses extractive reading comprehension on SQuAD, where a system must answer a question by selecting a contiguous span of a given passage. Such answers are often longer phrases, not single entities. The authors propose gated self-matching networks, an end-to-end model with four stages:

  1. Bi-directional GRU encoders for the question and the passage, using word and character embeddings.
  2. A gated attention-based recurrent layer that fuses question information into each passage word and filters it by relevance.
  3. A self-matching layer in which the passage attends over itself, so each word can gather evidence from distant parts of the text.
  4. A pointer network that predicts where the answer starts and ends.

On the hidden SQuAD test set, a single model reaches 71.3% exact match and an ensemble reaches 75.9%. Both held first place on the SQuAD leaderboard when the paper was submitted. The work matters because it shows that relevance gating combined with passage-wide self-attention addresses the limited memory of recurrent encoders in span-based QA.

Key points

  • Evaluated on SQuAD (Rajpurkar et al., 2016): 71.3% exact match for the single model and 75.9% for the ensemble on the hidden test set. Both ranked first on the leaderboard at submission time (Feb. 6, 2017).
  • The approach builds on match-LSTM and pointer networks (Wang and Jiang, 2016b). Its novelty is an additional sigmoid gate on the RNN input, which is the passage word concatenated with its attention-pooled question vector.
  • GRU was chosen over LSTM because it performs similarly at lower computational cost. The authors report that the extra gate helps with both cell types.
  • Character-level embeddings come from the final hidden states of a bi-directional RNN run over a token's characters, which helps with OOV words.
  • Training minimizes the summed negative log probabilities of the gold start and end positions. MS MARCO is named as the next target dataset.

Interesting ideas

  • I1. Relevance gate on attention input: a learned sigmoid gate over the concatenation of a token and its attended context can down-weight irrelevant tokens before recurrence. This filtering idea could carry over to any cross-sequence matching task, such as entailment or retrieval-based QA.
  • I2. Self-matching after cross-matching: the authors first make the passage question-aware and then let it attend to itself. This lets the model use clues far from a candidate answer, which is useful wherever RNN context is too short, such as long documents or multi-sentence evidence.
  • I3. Question-summary initial state for the pointer: initializing the pointer network with an attention-pooled question vector, built with a trainable query parameter, conditions span prediction on the question. This is a cheap trick for any boundary or span extraction decoder.
  • I4. Character-level RNN embeddings next to word vectors: combining the two gives robustness to rare and unseen words. This could be especially helpful for morphologically rich languages such as Ukrainian.
  • I5. Ablating a component across cell types: testing the added gate on both GRU and LSTM shows the gain is not tied to one architecture. This is a good evaluation habit for any proposed module.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

157 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 142 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1018

@inproceedings{wang2017gated,
  title = {{Gated Self-Matching Networks for Reading Comprehension and Question Answering}},
  author = {Wang, Wenhui and Yang, Nan and Wei, Furu and Chang, Baobao and Zhou, Ming},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1018},
  url = {https://doi.org/10.18653/v1/p17-1018}
}

References

33 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR . find our card
  2. Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. find our card
  3. Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D. Manning. 2014. Modeling biological processes for reading comprehension. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2… find our card
  4. Kenton Lee, Tom Kwiatkowski, Ankur Parikh, and Dipanjan Das. 2016. Learning recurrent span representations for extractive question answering. arXiv preprint arXiv:1611.01436 . — [arXiv:1611.01436](https://arxiv.org/abs/1611.01436) find
  5. Wang Ling, Yulia Tsvetkov, Silvio Amir, Ramon Fermandez, Chris Dyer, Alan W. Black, Isabel Trancoso, and Chu-Cheng Lin. 2015. Not all contexts are created equal: Better word representations with variable attention. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Proces… find
  6. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. find our card
  7. Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016. find our card
  8. Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The stanford corenlp natural language processing toolkit. In ACL (System Demonstrations). pages 55–60. find our card
  9. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Interspeech. find
  10. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan… find
  11. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR abs/1611.09268. find our card
  12. Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada. pages 2692–2700. find
  13. Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016. find our card
  14. Shuohang Wang and Jing Jiang. 2016a. Learning natural language inference with LSTM. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016. find our card
  15. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group… find
  16. Shuohang Wang and Jing Jiang. 2016b. Machine comprehension using match-lstm and answer pointer. arXiv preprint arXiv:1608.07905 . find our card
  17. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  18. Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. 2016. Multi-perspective context matching for machine comprehension. arXiv preprint arXiv:1612.04211 . — [arXiv:1612.04211](https://arxiv.org/abs/1612.04211) find
  19. Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Fastqa: A simple and efficient neural architecture for question answering. arXiv preprint arXiv:1703.04816 . find
  20. Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. pages 193–203. find our card
  21. Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. arXiv preprint arXiv:1611.01604 . find our card
  22. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. CoRR . find our card
  23. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of EMNLP. Citeseer, pages 2013–2018. find our card
  24. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603 . find our card
  25. Zhilin Yang, Bhuwan Dhingra, Ye Yuan, Junjie Hu, William W. Cohen, and Ruslan Salakhutdinov. 2016. Words or characters? fine-grained gating for reading comprehension. CoRR abs/1611.01724. — [arXiv:1611.01724](https://arxiv.org/abs/1611.01724) find
  26. Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2016. Reasonet: Learning to stop reading in machine comprehension. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Proce… find
  27. Yang Yu, Wei Zhang, Kazi Hasan, Mo Yu, Bing Xiang, and Bowen Zhou. 2016. End-to-end reading comprehension with dynamic answer chunk ranking. arXiv preprint arXiv:1610.09996 . — [arXiv:1610.09996](https://arxiv.org/abs/1610.09996) find
  28. Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. CoRR abs/1212.5701. find our card
  29. Junbei Zhang, Xiaodan Zhu, Qian Chen, Lirong Dai, and Hui Jiang. 2017. Exploring question understanding and adaptation in neural-network-based question answering. arXiv preprint arXiv:1703.04617 . — [arXiv:1703.04617](https://arxiv.org/abs/1703.04617) find
  30. Alessandro Sordoni, Phillip Bachman, and Yoshua Bengio. 2016. Iterative alternating neural attention for machine reading. CoRR abs/1606.02245. — [arXiv:1606.02245](https://arxiv.org/abs/1606.02245) find
  31. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research . find
  32. Adam Trischler, Zheng Ye, Xingdi Yuan, and Kaheer Suleman. 2016. Natural language comprehension with the epireader. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  33. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/