Elamur Research
☰
home / papers / 2016 / Bidirectional Attention Flow for Machine Comprehension

Bidirectional Attention Flow for Machine Comprehension

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1611.01603

Summary

Machine comprehension means answering a question using a given passage, which requires capturing how the question and passage relate. Earlier attention-based readers usually compressed the passage into one fixed-size vector early, made each attention step depend on the previous one, and attended in only one direction. This paper introduces BiDAF, a six-layer network. It builds word representations from character-level CNNs, pretrained GloVe vectors and bidirectional LSTMs. A shared similarity matrix then drives attention both from context to query and from query to context. Attention is computed separately at every passage position, does not depend on earlier steps, and its output is passed on with the earlier embeddings instead of being collapsed into one vector. LSTM modeling layers then predict where the answer span starts and ends. When submitted, BiDAF topped the SQuAD leaderboard. With only its output layer changed, it also achieved state-of-the-art results on the CNN/DailyMail cloze test. The design became a standard baseline for span-extraction question answering and can be adapted to QA datasets in other languages.

Key points

  • The model has six layers: character embedding (Char-CNN), word embedding (GloVe) passed through a two-layer Highway Network, contextual BiLSTM, attention flow, a two-layer BiLSTM modeling layer, and a task-specific output layer.
  • For each context word, the attention output joins the context vector, the attended query vector and their element-wise products into an 8d-dimensional feature.
  • A single BiDAF model scores 68.0 EM / 77.3 F1 on the SQuAD dev set. In the paper's comparison of similarity functions, the trainable similarity function beats a dot product (65.5 / 75.5), a bilinear form (61.6 / 71.8) and a purely linear form (59.5 / 69.7).
  • The appendix error analysis puts SQuAD mistakes into six categories, including imprecise answer boundaries, syntactic ambiguity, paraphrasing, missing external knowledge, multi-sentence reasoning and preprocessing faults.
  • Code and a demo were released at allenai.github.io/bi-att-flow.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

511 papers of our library cite it (of the 140673 whose bibliographies we read); 43 of them have a page here, 468 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1611.01603 · arXiv 1611.01603

@misc{seo2016bidirectional,
  title = {{Bidirectional Attention Flow for Machine Comprehension}},
  author = {Seo, Minjoon and Kembhavi, Aniruddha and Farhadi, Ali and Hajishirzi, Hannaneh},
  year = {2016},
  doi = {10.48550/arxiv.1611.01603},
  eprint = {1611.01603},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1611.01603}
}

References

33 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015. find our card
  3. Danqi Chen, Jason Bolton, and Christopher D. Manning. A thorough examination of the cnn/daily mail reading comprehension task. In ACL, 2016. find our card
  4. Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. Attention-overattention neural networks for reading comprehension. arXiv preprint arXiv:1607.04423, 2016. — [arXiv:1607.04423](https://arxiv.org/abs/1607.04423) find
  5. Bhuwan Dhingra, Hanxiao Liu, William W Cohen, and Ruslan Salakhutdinov. Gated-attention readers for text comprehension. arXiv preprint arXiv:1606.01549, 2016. find
  6. Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In EMNLP, 2016. find
  7. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, 2015. find our card
  8. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. The goldilocks principle: Reading children’s books with explicit memory representations. In ICLR, 2016. find our card
  9. Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 1997. find
  10. Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. Text understanding with the attention sum reader network. In ACL, 2016. find
  11. Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, 2014. find our card
  12. Sosuke Kobayashi, Ran Tian, Naoaki Okazaki, and Kentaro Inui. Dynamic entity representation with max-pooling improves machine reading. In NAACL-HLT, 2016. find
  13. Kenton Lee, Tom Kwiatkowski, Ankur Parikh, and Dipanjan Das. Learning recurrent span representations for extractive question answering. arXiv preprint arXiv:1611.01436, 2016. — [arXiv:1611.01436](https://arxiv.org/abs/1611.01436) find
  14. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In NIPS, 2016. find
  15. Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A neural-based approach to answering questions about images. In ICCV, 2015. find
  16. Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014. find
  17. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016. find our card
  18. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. Mctest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP, 2013. find our card
  19. Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. Reasonet: Learning to stop reading in machine comprehension. arXiv preprint arXiv:1609.05284, 2016. — [arXiv:1609.05284](https://arxiv.org/abs/1609.05284) find
  20. Alessandro Sordoni, Phillip Bachman, and Yoshua Bengio. Iterative alternating neural attention for machine reading. arXiv preprint arXiv:1606.02245, 2016. — [arXiv:1606.02245](https://arxiv.org/abs/1606.02245) find
  21. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014. find
  22. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387, 2015. find our card
  23. Adam Trischler, Zheng Ye, Xingdi Yuan, and Kaheer Suleman. Natural language comprehension with the epireader. In EMNLP, 2016. find
  24. Shuohang Wang and Jing Jiang. Machine comprehension using match-lstm and answer pointer. arXiv preprint arXiv:1608.07905, 2016. find our card
  25. Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In ICLR, 2015. find
  26. Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In ICML, 2016a. find
  27. Caiming Xiong, Victor Zhong, and Richard Socher. Dynamic coattention networks for question answering. arXiv preprint arXiv:1611.01604, 2016b. find our card
  28. Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In ECCV, 2016. find
  29. Zhilin Yang, Bhuwan Dhingra, Ye Yuan, Junjie Hu, William W Cohen, and Ruslan Salakhutdinov. Words or characters? fine-grained gating for reading comprehension. arXiv preprint arXiv:1611.01724, 2016. — [arXiv:1611.01724](https://arxiv.org/abs/1611.01724) find
  30. Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. arXiv preprint arXiv:1511.02274, 2015. find
  31. Yang Yu, Wei Zhang, Kazi Hasan, Mo Yu, Bing Xiang, and Bowen Zhou. End-to-end reading comprehension with dynamic answer chunk ranking. arXiv preprint arXiv:1610.09996, 2016. — [arXiv:1610.09996](https://arxiv.org/abs/1610.09996) find
  32. Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012. find our card
  33. Yuke Zhu, Oliver Groth, Michael S. Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In CVPR, 2016. find our card