Elamur Research⌕
☰
home / papers / 2015 / Stacked Attention Networks for Image Question Answering

Stacked Attention Networks for Image Question Answering

Paper · 2015 · Zichao Yang, Xiaodong He, Jianfeng Gao et al.

Summary

The paper addresses image question answering (QA): answering a natural-language question about a given picture. Earlier neural approaches merged a single global CNN image vector with a question encoding. That design struggles when the answer depends on small, specific parts of the scene. The authors argue that such questions need several reasoning steps, so they introduce stacked attention networks (SANs). VGGNet features from the last pooling layer split each image into a 14×14 grid of region vectors. An LSTM or a CNN with unigram, bigram and trigram filters encodes the question. The question vector then serves as a query that weights the image regions. The attended visual summary is added to the query, producing a refined query for the next attention layer, so the focus narrows step by step. SANs beat prior systems on four image QA benchmarks, including COCO-QA and VQA. Attention maps show the first layer spreading over the objects the question mentions and the second layer concentrating on the region holding the answer. The work showed that iterative visual attention is effective for multimodal reasoning.

Key points

  • Images are resized to 448×448. The last VGGNet pooling layer then yields 512×14×14 features, one vector per 32×32-pixel region, which a tanh layer projects to the question-vector dimension.
  • On the official VQA server, SAN(2, CNN) reaches 58.7% overall (79.3 Yes/No, 36.6 Number, 46.1 Other). This is 4.8 points above the strongest baseline, LSTM Q+I, with the largest gain (9.7 points) on the Other question type.
  • On COCO-QA, SAN(2, CNN) improves over the IMG+BOW and 2-VIS+BLSTM baselines by 7.2 points on Color, 6.1 on Objects, 5.7 on Location and 4.2 on Number.
  • Two attention layers beat one on all four data sets. Yes/No questions benefit little, since their answers depend mostly on the question text.
  • A manual check of 100 COCO-QA errors found these causes:
    • 22%: attention on the wrong region;
    • 42%: correct region but wrong answer;
    • 31%: ambiguous but plausible answers;
    • 5%: incorrect gold labels.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🏢MicrosoftRedmond · United States325 works · 142,931 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 441 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Question answering · 53 works

Cite

Cite as Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng and Alex Smola. 2015. Stacked Attention Networks for Image Question Answering. arXiv:1511.02274.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.02274 · arXiv 1511.02274

@misc{yang2015stacked,
  title = {{Stacked Attention Networks for Image Question Answering}},
  author = {Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng and Alex Smola},
  year = {2015},
  doi = {10.48550/arxiv.1511.02274},
  eprint = {1511.02274},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.02274}
}

References

32 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. arXiv preprint arXiv:1505.00468, 2015. 1, 2, 5, 6, 7 find
  2. D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. 1
  3. J. Berant and P. Liang. Semantic parsing via paraphrasing. In Proceedings of ACL, volume 7, page 92, 2014. 1
  4. A. Bordes, S. Chopra, and J. Weston. Question answering with subgraph embeddings. arXiv preprint arXiv:1406.3676, 2014. 1 find
  5. X. Chen and C. L. Zitnick. Learning a recurrent visual representation for image caption generation. arXiv preprint arXiv:1411.5654, 2014. 2 find
  6. H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. Platt, et al. From captions to visual concepts and back. arXiv preprint arXiv:1411.4952, 2014. 2, 3, 5 find
  7. H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu. Are you talking to a machine? dataset and methods for multilingual image question answering. arXiv preprint arXiv:1505.05612, 2015. 1, 2 find
  8. A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. 5 find
  9. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 1 find
  10. A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. arXiv preprint arXiv:1412.2306, 2014. 2
  11. Y. Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882, 2014. 3
  12. R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014. 2
  13. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012. 2
  14. A. Kumar, O. Irsoy, J. Su, J. Bradbury, R. English, B. Pierce, P. Ondruska, I. Gulrajani, and R. Socher. Ask me anything: Dynamic memory networks for natural language processing. arXiv preprint arXiv:1506.07285, 2015. 1
  15. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. 1 find
  16. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014, pages 740–755. Springer, 2014. 5
  17. L. Ma, Z. Lu, and H. Li. Learning to answer questions from image using convolutional neural network. arXiv preprint arXiv:1506.00333, 2015. 2, 5, 6 find
  18. M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In Advances in Neural Information Processing Systems, pages 1682–1690, 2014. 1, 2, 4, 5, 6 find
  19. M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions about images. arXiv preprint arXiv:1505.01121, 2015. 1, 2, 5, 6 — [arXiv:1505.01121](https://arxiv.org/abs/1505.01121) find
  20. J. Mao, W. Xu, Y. Yang, J. Wang, and A. Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint arXiv:1412.6632, 2014. 2 find
  21. M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. arXiv preprint arXiv:1505.02074, 2015. 1, 2, 5, 6 find
  22. Y. Shen, X. He, J. Gao, L. Deng, and G. Mesnil. A latent semantic model with convolutional-pooling structure for information retrieval. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pages 101–110. ACM, 2014. 3 find
  23. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 2
  24. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014. 5 find
  25. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112, 2014. 3
  26. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. 2 find
  27. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. arXiv preprint arXiv:1411.4555, 2014. 2
  28. J. Weston, S. Chopra, and A. Bordes. Memory networks. arXiv preprint arXiv:1410.3916, 2014. 1 find
  29. Z. Wu and M. Palmer. Verbs semantics and lexical selection. In Proceedings of the 32nd annual meeting on Association for Computational Linguistics, pages 133–138. Association for Computational Linguistics, 1994. 5 find
  30. K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044, 2015. 1, 2
  31. W.-t. Yih, M.-W. Chang, X. He, and J. Gao. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of the Joint Conference of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing of the AFNLP,…
  32. W.-t. Yih, X. He, and C. Meek. Semantic parsing for single-relation question answering. In Proceedings of ACL, 2014. 1 Figure 7: More examples

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2015-stacked-attention-networks-image-question/