Elamur Research
☰
home / papers / 2018 / QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1804.09541

Summary

Reading comprehension systems of the time, such as BiDAF, combined recurrent networks with attention. This made them slow to train and run because tokens are processed one after another. QANet, from Carnegie Mellon University and Google Brain, removes recurrence entirely. Its encoders pair depthwise separable convolutions, which capture local structure, with multi-head self-attention, which models relations between any two words. A standard context-query attention layer links the paragraph and the question. A stack of model encoder blocks then refines that representation, and the output layer predicts the start and end positions of the answer span. Because computation runs in parallel, the model trains 3–13x faster and infers 4–9x faster than RNN counterparts on SQuAD at comparable accuracy. The authors use the saved compute to train on more data: they paraphrase training examples by back-translation, passing English text through a neural machine translation model into another language and back. With this augmentation, a single model reaches 84.6 F1 on the SQuAD test set, compared with the previous best published 81.8. This shows that recurrence-free readers can be both fast and accurate.

Key points

  • Speed versus BiDAF: QANet reaches BiDAF's 77.0 F1 after about 3 hours of training instead of 15. With 18 hours of training it reaches 82.7 F1 on the SQuAD dev set.
  • Architecture details:
    • Inputs are fixed 300-d GloVe vectors plus 200-d character embeddings, followed by a two-layer highway network.
    • Encoder blocks use kernel size 7, d = 128, 8 attention heads, and layernorm with a residual connection around every sub-layer.
    • The embedding encoder is 1 block with 4 conv layers. The model encoder is 7 blocks with 2 conv layers each, with weights shared across its 3 passes.
  • Attention layer: context-query attention uses the trilinear similarity function, plus DCN-style query-to-context attention.
  • Ablations: combining convolution with self-attention gives +2.7 F1 over self-attention alone. Stochastic depth (layer dropout) adds about +0.2 F1.
  • Later leaderboard results (April 2018):
    • Single model: EM/F1 = 82.2/88.6.
    • Ensemble: EM/F1 = 83.9/89.7. Both ranked No.1, and the ensemble's EM exceeds the human score of 82.3.
    • The concurrent SAN model reported 84.4 F1, on par with QANet's 84.6.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

213 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 198 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1804.09541 · arXiv 1804.09541

@misc{yu2018qanet,
  title = {{QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension}},
  author = {Yu, Adams Wei and Dohan, David and Luong, Minh-Thang and Zhao, Rui and Chen, Kai and Norouzi, Mohammad and Le, Quoc V.},
  year = {2018},
  doi = {10.48550/arxiv.1804.09541},
  eprint = {1804.09541},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1804.09541}
}

References

51 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh… find our card
  2. Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.06450, 2016. URL http://arxiv.org/abs/1607.06450. ↗ our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015. find our card
  4. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pp. 1870–1879, 2017. find our card
  5. François Chollet. Xception: Deep learning with depthwise separable convolutions. abs/1610.02357, 2016. URL http://arxiv.org/abs/1610.02357. — [arXiv:1610.02357](https://arxiv.org/abs/1610.02357) ↗
  6. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. find
  7. Christopher Clark and Matt Gardner. Simple and effective multi-paragraph reading comprehension. CoRR, abs/1710.10723, 2017. URL http://arxiv.org/abs/1710.10723. ↗ our card
  8. Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. Attention-overattention neural networks for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Pape… find
  9. Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. Learning to paraphrase for question answering. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 875–886. Association for Computational Linguistics, 2017. URL http://aclweb.org/anthology/D17-109… ↗
  10. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In International Conference on Machine Learning, 2017. find our card
  11. Yichen Gong and Samuel R. Bowman. Ruminating reader: Reasoning with gated multi-hop attention. CoRR, abs/1704.07415, 2017. URL http://arxiv.org/abs/1704.07415. ↗
  12. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12,… find our card
  13. Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. Wikireading: A novel large-scale language understanding task over wikipedia. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics,… find
  14. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. The goldilocks principle: Reading children’s books with explicit memory representations. CoRR, abs/1511.02301, 2015. find our card
  15. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
  16. Minghao Hu, Yuxing Peng, and Xipeng Qiu. Reinforced mnemonic reader for machine comprehension. CoRR, abs/1705.02798, 2017. URL http://arxiv.org/abs/1705.02798. ↗
  17. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, pp. 646–661, 2016. find
  18. CoRR, Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp. 2021–2031, 2017. find our card
  19. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volu… find our card
  20. Lukasz Kaiser, Aidan N Gomez, and Francois Chollet. Depthwise separable convolutions for neural machine translation. arXiv preprint arXiv:1706.03059, 2017. — [arXiv:1706.03059](https://arxiv.org/abs/1706.03059) find
  21. Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1746– 1751, 2014. find our card
  22. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. abs/1412.6980, 2014. URL http://arxiv.org/abs/1412.6980. ↗
  23. Kenton Lee, Tom Kwiatkowski, Ankur P. Parikh, and Dipanjan Das. Learning recurrent span representations for extractive question answering. CoRR, abs/1611.01436, 2016. — [arXiv:1611.01436](https://arxiv.org/abs/1611.01436) find
  24. Rui Liu, Junjie Hu, Wei Wei, Zi Yang, and Eric Nyberg. Structural embedding of syntactic trees for machine comprehension. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp. 826–835, 2017a. find
  25. Xiaodong Liu, Yelong Shen, Kevin Duh, and Jianfeng Gao. Stochastic answer networks for machine reading comprehension. CoRR, abs/1712.03556, 2017b. URL http://arxiv.org/abs/1712.03556. ↗
  26. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In EMNLP, 2015. find
  27. Minh-Thang Luong, Eugene Brevdo, and Rui Zhao. Neural machine translation (seq2seq) tutorial. https://github.com/tensorflow/nmt, 2017. ↗
  28. Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. Paraphrasing revisited with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pp. 881–893. Association for Computational Linguistics… find
  29. Boyuan Pan, Hao Li, Zhou Zhao, Bin Cao, Deng Cai, and Xiaofei He. MEMEN: multi-layer embedding with memory networks for machine comprehension. CoRR, abs/1707.09098, 2017. — [arXiv:1707.09098](https://arxiv.org/abs/1707.09098) find
  30. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word//w representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543, 2014. URL http://www.aclweb.org/anthology/D14-1162. ↗
  31. Jonathan Raiman and John Miller. Globally normalized reader. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp. 1070–1080, 2017. find
  32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 2383–2392, 2016. find our card
  33. Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In ACL (1). The Association for Computer Linguistics, 2016. find
  34. CoRR, Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. CoRR, abs/1611.01603, 2016. URL http://arxiv.org/abs/1611.01603. ↗ our card
  35. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. Disan: Directional self-attention network for rnn/cnn-free language understanding. CoRR, abs/1709.04696, 2017a. URL http://arxiv.org/abs/1709.04696. ↗
  36. Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. Reasonet: Learning to stop reading in machine comprehension. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017, pp. 1047–1055, 2017b. find
  37. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. CoRR, abs/1505.00387, 2015. URL http://arxiv.org/abs/1505.00387. ↗ our card
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017a. URL http://arxiv.org/abs/1706.03762. ↗
  39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017b. find
  40. Shuohang Wang and Jing Jiang. Machine comprehension using match-lstm and answer pointer. CoRR, abs/1608.07905, 2016. URL http://arxiv.org/abs/1608.07905. ↗ our card
  41. Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Pap… find
  42. Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. Multi-perspective context matching for machine comprehension. CoRR, abs/1612.04211, 2016. URL http://arxiv.org/abs/1612.04211. — [arXiv:1612.04211](https://arxiv.org/abs/1612.04211) ↗
  43. Dirk Weissenborn, Georg Wiese, and Laura Seiffe. Making neural QA as simple as possible but not simpler. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Vancouver, Canada, August 3-4, 2017, pp. 271–280, 2017. find
  44. John Wieting, Jonathan Mallinson, and Kevin Gimpel. Learning paraphrastic sentence embeddings from back-translated bitext. In EMNLP, pp. 274–285. Association for Computational Linguistics, 2017. find
  45. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  46. Caiming Xiong, Victor Zhong, and Richard Socher. Dynamic coattention networks for question answering. CoRR, abs/1611.01604, 2016. URL http://arxiv.org/abs/1611.01604. ↗ our card
  47. Adams Wei Yu, Hongrae Lee, and Quoc V. Le. Learning to skim text. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pp. 1880–1890, 2017. find
  48. Yang Yu, Wei Zhang, Kazi Saidul Hasan, Mo Yu, Bing Xiang, and Bowen Zhou. End-to-end reading comprehension with dynamic answer chunk ranking. CoRR, abs/1610.09996, 2016. URL http: //arxiv.org/abs/1610.09996. — [arXiv:1610.09996](https://arxiv.org/abs/1610.09996) find
  49. Junbei Zhang, Xiao-Dan Zhu, Qian Chen, Li-Rong Dai, Si Wei, and Hui Jiang. Exploring question understanding and adaptation in neural-network-based question answering. CoRR, abs/1703.04617, 2017. — [arXiv:1703.04617](https://arxiv.org/abs/1703.04617) find
  50. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 649–657, 2015. find our card
  51. Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. Neural question generation from text: A preliminary study. CoRR, abs/1704.01792, 2017. URL http: //arxiv.org/abs/1704.01792. — [arXiv:1704.01792](https://arxiv.org/abs/1704.01792) find