Elamur Research
☰
home / papers / 2016 / Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1612.00837

Summary

VQA models can score well by relying on regularities in the questions instead of looking at the image. In the original VQA dataset, some question openings nearly decide the answer by themselves. The authors fix this by rebalancing VQA question by question. For each (image, question, answer) triplet, Amazon Mechanical Turk workers looked at 24 visually similar COCO images, retrieved with VGGNet fc7 features. From these they picked a complementary image where the question still applies but the answer is different. Ten more workers then answered the question for that new image. The result, released as VQA v2.0, has about twice as many image-question pairs as the original. Nearly every question now comes with two very similar images that need different answers. The authors tested several state-of-the-art models, including the winner of the 2016 VQA Challenge. Every one of them lost accuracy on the balanced data. This is the first concrete evidence that these models had been exploiting language priors. The same collection protocol also made it possible to build an explainable model that supports its answer by retrieving a counter-example image.

Key points

  • The bias is extreme in the original VQA dataset. Answering "yes" to every question that starts with "Do you see a…" scores 87% accuracy. "tennis" is correct for 41% of "What sport is" questions.
  • Balancing targets the conditional answer distribution P(A|Q), not only the overall P(A). This way the image itself has to decide the answer.
  • VQA v2.0 contains about 1.1M (image, question) pairs and about 13M answers over roughly 200k COCO images. It is available at visualqa.org.
  • Benchmarked models: a baseline model, an attention-based model, and the VQA Real Open-Ended Challenge 2016 winner, compared against a language-only model. All of them perform significantly worse on the balanced set.
  • The counter-example explanation model has an answering head and a hinge-ranking explaining head. On Recall@5 it scores 43.39, compared with 42.84 for the nearest-image Distance baseline, 21.65 for the VQA-model baseline and 20.79 for Random.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

692 papers of our library cite it (of the 140673 whose bibliographies we read); 60 of them have a page here, 632 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1612.00837 · arXiv 1612.00837

@misc{goyal2016making,
  title = {{Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering}},
  author = {Goyal, Yash and Khot, Tejas and Summers-Stay, Douglas and Batra, Dhruv and Parikh, Devi},
  year = {2016},
  doi = {10.48550/arxiv.1612.00837},
  eprint = {1612.00837},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1612.00837}
}

References

49 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Agrawal, D. Batra, and D. Parikh. Analyzing the Behavior of Visual Question Answering Models. In EMNLP, 2016. 1 find
  2. J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Deep compositional question answering with neural module networks. In CVPR, 2016. 2 find
  3. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. VQA: Visual Question Answering. In ICCV, 2015. 1, 2, 4, 5, 6, 9, 10 find
  4. X. Chen and C. L. Zitnick. Mind’s Eye: A Recurrent Visual Representation for Image Caption Generation. In CVPR, 2015. 1 find
  5. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009. 9 find
  6. J. Devlin, S. Gupta, R. B. Girshick, M. Mitchell, and C. L. Zitnick. Exploring nearest neighbor approaches for image captioning. CoRR, abs/1505.04467, 2015. 1 — [arXiv:1505.04467](https://arxiv.org/abs/1505.04467) find
  7. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term Recurrent Convolutional Networks for Visual Recognition and Description. In CVPR, 2015. 1 find our card
  8. H. Fang, S. Gupta, F. N. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt, C. L. Zitnick, and G. Zweig. From Captions to Visual Concepts and Back. In CVPR, 2015. 1 find
  9. A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding. In EMNLP, 2016. 2, 6, 7 find
  10. H. Gao, J. Mao, J. Zhou, Z. Huang, and A. Yuille. Are you talking to a machine? dataset and methods for multilingual image question answering. In NIPS, 2015. 1, 2 find
  11. Y. Goyal, A. Mohapatra, D. Parikh, and D. Batra. Towards Transparent AI Systems: Interpreting Visual Question Answering Models. In ICML Workshop on Visualization for Deep Learning, 2016. 4 find
  12. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016. 6 find
  13. L. A. Hendricks, Z. Akata, M. Rohrbach, J. Donahue, B. Schiele, and T. Darrell. Generating visual explanations. In ECCV, 2016. 4 find
  14. M. Hodosh and J. Hockenmaier. Focused evaluation for image description with binary forced-choice tasks. In Workshop on Vision and Language, Annual Meeting of the Association for Computational Linguistics, 2016. 3 find
  15. I. Ilievski, S. Yan, and J. Feng. A focused dynamic attention model for visual question answering. CoRR, abs/1604.01485, 2016. 2 — [arXiv:1604.01485](https://arxiv.org/abs/1604.01485) find
  16. A. Jabri, A. Joulin, and L. van der Maaten. Revisiting Visual Question Answering Baselines. In ECCV, 2016. 1 find
  17. K. Kafle and C. Kanan. Answer-type prediction for visual question answering. In CVPR, 2016. 2 find
  18. K. Kafle and C. Kanan. Visual Question Answering: Datasets, Algorithms, and Future Challenges. CoRR, abs/1610.01465, 2016. 1 — [arXiv:1610.01465](https://arxiv.org/abs/1610.01465) find
  19. A. Karpathy and L. Fei-Fei. Deep Visual-Semantic Alignments for Generating Image Descriptions. In CVPR, 2015. 1 find our card
  20. J.-H. Kim, S.-W. Lee, D.-H. Kwak, M.-O. Heo, J. Kim, J.-W. Ha, and B.-T. Zhang. Multimodal Residual Learning for Visual QA. In NIPS, 2016. 2 find
  21. R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models. TACL, 2015. 1 find our card
  22. R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. arXiv preprint arXiv:1602.07332, 2016. 2 find our card
  23. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common Objects in Context. In ECCV, 2014. 2, 4 find our card
  24. J. Lu, X. Lin, D. Batra, and D. Parikh. Deeper LSTM and normalized CNN Visual Question Answering model. https://github.com/VT-vision-lab/VQA\_LSTM\_CNN, 2015. 2, 6, 7, 8 ↗
  25. J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical Question-Image Co-Attention for Visual Question Answering. In NIPS, 2016. 2, 6, 7 find
  26. M. Malinowski and M. Fritz. A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input. In NIPS, 2014. 1, 2 find
  27. M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions about images. In ICCV, 2015. 1, 2 find
  28. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Explain Images with Multimodal Recurrent Neural Networks. In NIPS, 2014. 1 find
  29. H. Noh and B. Han. Training recurrent answering units with joint loss minimization for vqa. CoRR, abs/1606.03647, 2016. 2 — [arXiv:1606.03647](https://arxiv.org/abs/1606.03647) find
  30. A. Ray, G. Christie, M. Bansal, D. Batra, and D. Parikh. Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions. In EMNLP, 2016. 8 find
  31. M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. In NIPS, 2015. 1, 2 find
  32. M. T. Ribeiro, S. Singh, and C. Guestrin. ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In Knowledge Discovery and Data Mining (KDD), 2016. 4 find our card
  33. K. Saito, A. Shin, Y. Ushiku, and T. Harada. Dualnet: Domain-invariant network for visual question answering. CoRR, abs/1606.06108, 2016. 2 — [arXiv:1606.06108](https://arxiv.org/abs/1606.06108) find
  34. R. R. Selvaraju, A. Das, R. Vedantam, M. Cogswell, D. Parikh, and D. Batra. Grad-CAM: Why did you say that? Visual Explanations from Deep Networks via Gradient-based Localization. arXiv preprint arXiv:1610.02391, 2016. 4 find our card
  35. K. J. Shih, S. Singh, and D. Hoiem. Where to look: Focus regions for visual question answering. In CVPR, 2016. 2 find
  36. A. Shin, Y. Ushiku, and T. Harada. The Color of the Cat is Gray: 1 Million Full-Sentences Visual Question Answering (FSVQA). arXiv preprint arXiv:1609.06657, 2016. 2 — [arXiv:1609.06657](https://arxiv.org/abs/1609.06657) find
  37. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015. 2, 4, 6 find our card
  38. M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler. MovieQA: Understanding Stories in Movies through Question-Answering. In CVPR, 2016. 2 find
  39. A. Torralba and A. Efros. Unbiased look at dataset bias. In CVPR, 2011. 2 find
  40. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and Tell: A Neural Image Caption Generator. In CVPR, 2015. 1 find our card
  41. P. Wang, Q. Wu, C. Shen, A. van den Hengel, and A. R. Dick. Explicit knowledge-based reasoning for visual question answering. CoRR, abs/1511.02570, 2015. 2 — [arXiv:1511.02570](https://arxiv.org/abs/1511.02570) find
  42. Q. Wu, P. Wang, C. Shen, A. van den Hengel, and A. R. Dick. Ask me anything: Free-form visual question answering based on knowledge from external sources. In CVPR, 2016. 2 find
  43. C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering. In ICML, 2016. 2 find
  44. H. Xu and K. Saenko. Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering. In ECCV, 2016. 2 find
  45. Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked Attention Networks for Image Question Answering. In CVPR, 2016. 2 find
  46. L. Yu, E. Park, A. C. Berg, and T. L. Berg. Visual Madlibs: Fill-in-the-blank Description Generation and Question Answering. In ICCV, 2015. 2 find
  47. P. Zhang, Y. Goyal, D. Summers-Stay, D. Batra, and D. Parikh. Yin and Yang: Balancing and Answering Binary Visual Questions. In CVPR, 2016. 1, 2, 4 find
  48. B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. In CVPR, 2015. 4 find
  49. B. Zhou, Y. Tian, S. Sukhbaatar, A. Szlam, and R. Fergus. Simple Baseline for Visual Question Answering. CoRR, abs/1512.02167, 2015. 1 — [arXiv:1512.02167](https://arxiv.org/abs/1512.02167) find