Elamur Research
☰
home / papers / 2016 / Dynamic Coattention Networks For Question Answering

Dynamic Coattention Networks For Question Answering

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1611.01604

Summary

The paper targets extractive question answering on SQuAD, where the answer is a contiguous span of a document. Earlier neural readers choose the span in a single pass, so an early wrong guess cannot be corrected. The authors propose the Dynamic Coattention Network (DCN), which has two parts. The first is a coattention encoder. A shared LSTM encodes both question and document, and an affinity matrix over all word pairs lets each text attend to the other. The resulting attention contexts are combined and passed through a bidirectional LSTM. The second part is a dynamic pointing decoder. It repeatedly re-estimates the start and end positions using an LSTM state and Highway Maxout Networks, which lets the model escape poor initial span choices. As a single model, DCN raised the best published SQuAD F1 from 71.0% to 75.9%. An ensemble reached 80.4% F1. This showed that refining answers iteratively and modeling question–document interaction jointly yield large gains in reading comprehension.

Key points

  • Encoder: The question and the document share one LSTM encoder, and each sequence gets a sentinel vector so the model can avoid attending to any specific word. The question encoding receives an extra non-linear projection.
  • Coattention: Attention weights are normalized row-wise and column-wise over the affinity matrix. Summaries are computed in both directions, including attention over the earlier attention contexts, and then fused with the document encoding by a bi-LSTM.
  • Decoder: Separate Highway Maxout Networks (HMN start, HMN end) score start and end positions. Training minimizes cross-entropy accumulated over all iterations. Decoding stops once the predictions no longer change or a maximum iteration count is reached.
  • Results: The single model scored 75.9% F1 versus 71.0% for Yu et al. (2016). The ensemble scored 80.4% F1, while the next-best entry on the SQuAD leaderboard (as of Nov. 3, 2016) had 78.1%.
  • Error analysis: Wrong answers usually still have the right type (e.g., a person), even though no type information is given to the model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

128 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 115 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1611.01604 · arXiv 1611.01604

@misc{xiong2016dynamic,
  title = {{Dynamic Coattention Networks For Question Answering}},
  author = {Xiong, Caiming and Zhong, Victor and Socher, Richard},
  year = {2016},
  doi = {10.48550/arxiv.1611.01604},
  eprint = {1611.01604},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1611.01604}
}

References

28 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2425–2433, 2015. find
  2. Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D Manning. Modeling biological processes for reading comprehension. In EMNLP, 2014. find
  3. Julian Besag. On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society. Series B (Methodological), pp. 259–302, 1986. find
  4. Danqi Chen, Jason Bolton, and Christopher D. Manning. A thorough examination of the cnn/daily mail reading comprehension task. In Association for Computational Linguistics (ACL), 2016. find our card
  5. Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. Attention-overattention neural networks for reading comprehension. arXiv preprint arXiv:1607.04423, 2016. — [arXiv:1607.04423](https://arxiv.org/abs/1607.04423) find
  6. Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron C Courville, and Yoshua Bengio. Maxout networks. ICML (3), 28:1319–1327, 2013. find
  7. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pp. 1693–1701, 2015. find our card
  8. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. The goldilocks principle: Reading children’s books with explicit memory representations. In ICLR, 2016. find our card
  9. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
  10. Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. Text understanding with the attention sum reader network. arXiv preprint arXiv:1603.01547, 2016. find
  11. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. find
  12. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. arXiv preprint arXiv:1606.00061, 2016. — [arXiv:1606.00061](https://arxiv.org/abs/1606.00061) find
  13. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412–1421. Association for Computational Linguistics, September 2015. find
  14. Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In ACL (System Demonstrations), pp. 55–60, 2014. find our card
  15. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. find our card
  16. Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pp. 1532–43, 2014. find
  17. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), 2016. find our card
  18. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. Mctest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP, volume 3, pp. 4, 2013. find our card
  19. Alessandro Sordoni, Phillip Bachman, and Yoshua Bengio. Iterative alternating neural attention for machine reading. arXiv preprint arXiv:1606.02245, 2016. — [arXiv:1606.02245](https://arxiv.org/abs/1606.02245) find
  20. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. find
  21. Rupesh K Srivastava, Klaus Greff, and Juergen Schmidhuber. Training very deep networks. In Advances in Neural Information Processing Systems 28, pp. 2377–2385, 2015. find
  22. Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. Chainer: a next-generation open source framework for deep learning. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS), 2015. find
  23. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information Processing Systems, pp. 2692–2700, 2015. find
  24. Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. Machine comprehension with syntax, frames, and semantics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short… find
  25. Shuohang Wang and Jing Jiang. Learning natural language inference with LSTM. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1442–1451. Association for Computational Linguistics, 2016a. find
  26. Shuohang Wang and Jing Jiang. Machine comprehension using match-LSTM and answer pointer. arXiv preprint arXiv:1608.07905, 2016b. find our card
  27. Y. Yu, W. Zhang, K. Hasan, M. Yu, B. Xiang, and B. Zhou. End-to-End Reading Comprehension with Dynamic Answer Chunk Ranking. ArXiv e-prints, October 2016. find
  28. Yang Yu, Wei Zhang, Kazi Hasan, Mo Yu, Bing Xiang, and Bowen Zhou. End-to-end answer chunk extraction and ranking for reading comprehension. arXiv preprint arXiv:1610.09996v2, 2016. — [arXiv:1610.09996](https://arxiv.org/abs/1610.09996) find