Elamur Research
☰
home / papers / 2015 / Visual7W: Grounded Question Answering in Images

Visual7W: Grounded Question Answering in Images

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1511.03416

Summary

Earlier visual question answering benchmarks tied each question and answer to the image as a whole. Many real questions, however, concern specific objects. Visual7W addresses this gap by linking the objects named in QA text to bounding boxes in the image. The authors used Amazon Mechanical Turk to collect multiple-choice QA pairs on COCO images, organized around seven question words. Six of them (what, where, when, who, why, how) are "telling" questions with text answers. The seventh, which, forms "pointing" questions whose answer is an image region. Crowd workers wrote the wrong answer options themselves, because automatically generated distractors often turned out to restate the correct answer. The paper also introduces an LSTM with spatial attention that picks out relevant image areas as it reads the question. Humans score far higher than every model tested, and the attention LSTM gives the best machine result. Grounding makes it possible to resolve ambiguous references and to inspect where a model looks. The resource is therefore useful for diagnosing how vision-and-language systems reason.

Key points

  • Dataset size: 327,939 QA pairs on 47,300 COCO images, with 1,311,756 human-written answer choices and 561,459 object groundings across 36,579 categories (about 12 boxes per image).
  • Format and quality control: Each question has four candidate answers. Three workers voted on every QA pair, and the overall acceptance rate was 85.8%, ranging from 92% for what down to 63% for why.
  • Accuracy: Humans reach 96.6%, a Question + Image LSTM baseline 52.1%, and the proposed LSTM-Att 55.6%; chance is 25%. Without the image, the LSTM (46.2%) beats humans (35.3%) by learning answer priors from training data.
  • Attention analysis: Answer-object boxes cover about 12% of the image area on average, yet the attention peak lands inside them 24% of the time. On pointing questions, accuracy stays above 50% even for rare object categories.
  • Relation to Visual Genome: Visual7W is a subset of Visual Genome, which holds 1.7 million 7W QA pairs. Visual7W adds groundings, answer choices and human evaluation on top of that collection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

139 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 122 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.03416 · arXiv 1511.03416

@misc{zhu2015visual7w,
  title = {{Visual7W: Grounded Question Answering in Images}},
  author = {Zhu, Yuke and Groth, Oliver and Bernstein, Michael and Fei-Fei, Li},
  year = {2015},
  doi = {10.48550/arxiv.1511.03416},
  eprint = {1511.03416},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.03416}
}

References

52 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. VQA: Visual question answering. ICCV, 2015. 1, 2, 3, 4, 5, 6 find
  2. K. Barnard, P. Duygulu, D. Forsyth, N. De Freitas, D. M. Blei, and M. I. Jordan. Matching words and pictures. The Journal of Machine Learning Research, 3:1107–1135, 2003. 2 find
  3. J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User Interface Software and Technology, 2010. 2 find
  4. X. Chen and C. L. Zitnick. Mind’s eye: A recurrent visual representation for image caption generation. In CVPR, 2015. 1, 2 find
  5. J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015. 1, 2, 5 find our card
  6. D. Ferrucci et al. Building Watson: An overview of the DeepQA project. AI Magazine, 2010. 2 find
  7. H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu. Are you talking to a machine? dataset and methods for multilingual image question answering. NIPS, 2015. 1, 2, 4, 5 find
  8. D. Geman, S. Geman, N. Hallonquist, and L. Younes. Visual turing test for computer vision systems. Proceedings of the National Academy of Sciences, 112(12):3618–3623, 2015. 1, 2, 4 find
  9. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. 2 find our card
  10. K. Gregor, I. Danihelka, A. Graves, and D. Wierstra. Draw: A recurrent neural network for image generation. arXiv preprint arXiv:1502.04623, 2015. 5 find
  11. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 5 find
  12. M. Iyyer, J. Boyd-Graber, L. Claudino, R. Socher, and H. Daumé III. A neural network for factoid question answering over paragraphs. In Empirical Methods in Natural Language Processing, 2014. 2 find
  13. A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. CVPR, 2015. 1, 2, 5 find our card
  14. A. Karpathy, A. Joulin, and L. Fei-Fei. Deep fragment embeddings for bidirectional image sentence mapping. In NIPS, pages 1889–1897, 2014. 1 find
  15. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014. 1, 2 find
  16. S. Kazemzadeh, V. Ordonez, M. Matten, and T. L. Berg. Referit game: Referring to objects in photographs of natural scenes. In EMNLP, 2014. 2 find
  17. M. H. Kiapour, X. Han, S. Lazebnik, A. C. Berg, and T. L. Berg. Where to buy it: Matching street clothing photos in online shops. ICCV, 2015. 2 find
  18. D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 6 find
  19. C. Kong, D. Lin, M. Bansal, R. Urtasun, and S. Fidler. What are you talking about? text-to-image coreference. In CVPR, 2014. 1, 2 find
  20. R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, M. Bernstein, and L. Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In arXiv preprint arxiv:1602.07332, 2016. 2 find our card
  21. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097–1105, 2012. 1 find
  22. R. Kuhn and E. Neveu. Political journalism: New challenges, new practices. Routledge, 2013. 2 find
  23. C. H. Lampert, H. Nickisch, and S. Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In CVPR, 2009. 2 find
  24. T.-Y. Lin, Y. Cui, S. Belongie, and J. Hays. Learning deep representations for ground-to-aerial geolocalization. In CVPR, 2015. 2 find
  25. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV. 2014. 1, 2, 4 find our card
  26. L. Ma, Z. Lu, and H. Li. Learning to answer questions from image using convolutional neural network. arXiv preprint arXiv:1506.00333, 2015. 2 — [arXiv:1506.00333](https://arxiv.org/abs/1506.00333) find
  27. M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, pages 1682–1690, 2014. 1, 2, 4, 5 find
  28. M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions about images. ICCV, 2015. 2, 5, 7 find
  29. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013. 7 find
  30. F. Palermo, J. Hays, and A. A. Efros. Dating historical color images. In ECCV. 2012. 2 find
  31. G. Patterson and J. Hays. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In CVPR, 2012. 2 find
  32. L. C. Pickup, Z. Pan, D. Wei, Y. Shih, C. Zhang, A. Zisserman, B. Scholkopf, and W. T. Freeman. Seeing the arrow of time. In CVPR, 2014. 2 find
  33. H. Pirsiavash, C. Vondrick, and A. Torralba. Inferring the why in images. arXiv preprint arXiv:1406.5472, 2014. 2 — [arXiv:1406.5472](https://arxiv.org/abs/1406.5472) find
  34. B. Plummer, L. Wang, C. Cervantes, J. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. ICCV, 2015. 1 find
  35. V. Ramanathan, A. Joulin, P. Liang, and L. Fei-Fei. Linking people with ”their” names using coreference resolution. In ECCV, 2014. 1, 2 find
  36. M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. NIPS, 2015. 1, 2, 4, 5 find
  37. M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal, and B. Schiele. Translating video content to natural language descriptions. In ICCV, 2013. 2 find
  38. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, pages 1–42, April 2015. 1, 4 find our card
  39. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2014. 1, 2, 5, 6 find our card
  40. R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2:207–218, 2014. 1, 2 find our card
  41. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112, 2014. 5 find
  42. Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface: Closing the gap to human-level performance in face verification. In CVPR, 2014. 1, 2 find
  43. A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deep neural networks. In CVPR, 2014. 1 find
  44. K. Tu, M. Meng, M. W. Lee, T. E. Choe, and S.-C. Zhu. Joint video and text parsing for understanding events and answering queries. In IEEE MultiMedia, 2014. 2 find
  45. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In CVPR, 2015. 1, 2 find our card
  46. J. Weston, A. Bordes, S. Chopra, and T. Mikolov. Towards ai-complete question answering: a set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015. 2, 4 find our card
  47. K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015. 1, 2, 5 find our card
  48. P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67– 78, 2014. 1 find
  49. L. Yu, E. Park, A. C. Berg, and T. L. Berg. Visual Madlibs: Fill in the blank Image Generation and Question Answering. ICCV, 2015. 1, 2, 4 find
  50. B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. In NIPS, 2014. 2 find
  51. Y. Zhu, A. Fathi, and L. Fei-Fei. Reasoning about object affordances in a knowledge base representation. ECCV, 2014. 2 find
  52. C. L. Zitnick, D. Parikh, and L. Vanderwende. Learning the visual interpretation of sentences. In ICCV, 2013. 2 find