Elamur Research
☰
home / papers / 2019 / GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/1902.09506

Summary

Popular VQA benchmarks let models score well by relying on language priors, such as typical object colours, rather than looking at the image. Their questions are also often shallow and come without structural annotation. GQA addresses this by generating questions automatically from Visual Genome scene graphs for 113K real photographs. The authors first clean and enrich the graphs. A question engine then walks over the objects, attributes and relations in each graph and combines them with hundreds of grammar patterns. The result is 22M compositional, multi-step questions. Each question comes with a functional program that encodes its meaning, plus a textual and a visual justification of the answer. The programs let the authors smooth answer distributions for both yes/no and open questions, which produces a balanced subset of 1.7M questions. They also support new metrics beyond accuracy. Tested baselines leave a wide gap: a question-only LSTM gets 42.1%, the best model (MAC) gets 54.1%, and humans reach 89.3%. GQA is useful as a realistic but controlled testbed for checking whether vision-language models actually reason.

Key points

  • Scale: 113K images from COCO and Flickr (via Visual Genome), 22M generated questions, and a balanced version of about 1.7M questions. It is available at visualreasoning.net.
  • Unlike VQA 2.0, which balances only binary questions, GQA uses a tunable smoothing method to even out answer distributions for open questions too.
  • New metrics: consistency, validity, plausibility, distribution (Chi-Square) and grounding. Attention-based models such as BottomUp and MAC score about 80% on grounding, but only 43% when given spatial instead of object-based features.
  • On the test set, Global Prior scores 28.9%, CNN+LSTM 46.6%, BottomUp 49.7%, MAC 54.1% and humans 89.3%. Even the strong models still give invalid or implausible answers 5–15% of the time.
  • Transfer: MAC trained on GQA reaches 52.1% on VQA without fine-tuning and 60.5% with it. MAC trained on VQA reaches only 39.8% and 46.5% on GQA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

546 papers of our library cite it (of the 140673 whose bibliographies we read); 38 of them have a page here, 508 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1902.09506 · arXiv 1902.09506

@misc{hudson2019gqa,
  title = {{GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering}},
  author = {Hudson, Drew A. and Manning, Christopher D.},
  year = {2019},
  doi = {10.48550/arxiv.1902.09506},
  eprint = {1902.09506},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1902.09506}
}

References

41 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Google books ngram corpus http://books.google.com/ngrams/. 17 ↗
  2. A. Agrawal, D. Batra, and D. Parikh. Analyzing the behavior of visual question answering models. In EMNLP, pages 1955–1960, 2016. 1, 2 find
  3. A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi. Don’t just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4971–4980, 2018. 1, 2 find
  4. A. Agrawal, J. Lu, S. Antol, M. Mitchell, C. L. Zitnick, D. Parikh, and D. Batra. VQA: Visual question answering. International Journal of Computer Vision, 123(1):4–31, 2017. 1, 2, 4 find
  5. P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and VQA. arXiv preprint arXiv:1707.07998, 2017. 1, 7, 8, 14, 15 — [arXiv:1707.07998](https://arxiv.org/abs/1707.07998) find
  6. J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 39–48, 2016. 1 find our card
  7. Y. Attali and M. Bar-Hillel. Guess where: The position of correct answers in multiple-choice test items as a psychometric variable. Journal of Educational Measurement, 40(2):109–128, 2003. 2 find
  8. A. Das, H. Agrawal, L. Zitnick, D. Parikh, and D. Batra. Human attention in visual question answering: Do humans and deep networks look at the same regions? Computer Vision and Image Understanding, 163:90–100, 2017. 1, 2 find
  9. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255. Ieee, 2009. 14 find
  10. A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016. 1 find
  11. Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6325–6334, 2017. 1, 2, 3, 6, 16 find our card
  12. D. A. Hudson and C. D. Manning. Compositional attention networks for machine reasoning. International Conference for Representation Learning (ICLR), 2018. 1, 6, 7, 8, 14, 15 find
  13. A. Jabri, A. Joulin, and L. van der Maaten. Revisiting visual question answering baselines. In European conference on computer vision, pages 727–739. Springer, 2016. 1 find
  14. U. Jain, Z. Zhang, and A. G. Schwing. Creativity: Generating diverse questions using variational autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5415–5424, 2017. 3 find
  15. J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1988–1997, 2017. 1, 2, 3, 7 find our card
  16. J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3668–3678, 2015. 3 find
  17. K. Kafle and C. Kanan. An analysis of visual question answering algorithms. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 1983–1991. IEEE, 2017. 2, 3 find
  18. K. Kafle and C. Kanan. Visual question answering: Datasets, algorithms, and future challenges. Computer Vision and Image Understanding, 163:3–20, 2017. 1, 2, 16 find
  19. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 15 find
  20. R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73, 2017. 1, 2, 3, 4, 17 find our card
  21. H. O. Lancaster and E. Seneta. Chi-square distribution. Encyclopedia of biostatistics, 2, 2005. 8 find
  22. Q. Li, Q. Tao, S. Joty, J. Cai, and J. Luo. VQA-E: Explaining, elaborating, and enhancing your answers for visual questions. arXiv preprint arXiv:1803.07464, 2018. 2 — [arXiv:1803.07464](https://arxiv.org/abs/1803.07464) find
  23. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 4, 17 find our card
  24. A. Mahendru, V. Prabhu, A. Mohapatra, D. Batra, and S. Lee. The promise of premise: Harnessing question premises in visual question answering. arXiv preprint arXiv:1705.00601, 2017. 3 find
  25. M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In Advances in neural information processing systems, pages 1682–1690, 2014. 3 find
  26. J. Millman, C. H. Bishop, and R. Ebel. An analysis of test-wiseness. Educational and Psychological Measurement, 25(3):707–726, 1965. 2 find
  27. J. J. Mondak and B. C. Davis. Asked and answered: Knowledge levels when we won’t take ‘don’t know’ for an answer. Political Behavior, 23(3):199–224, 2001. 2 find
  28. N. Mostafazadeh, I. Misra, J. Devlin, M. Mitchell, X. He, and L. Vanderwende. Generating natural questions about an image. arXiv preprint arXiv:1603.06059, 2016. 3 find
  29. D. H. Park, L. A. Hendricks, Z. Akata, A. Rohrbach, B. Schiele, T. Darrell, and M. Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 2 find
  30. J. Pennington, R. Socher, and C. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014. 14 find
  31. S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015. 14, 16, 17 find our card
  32. Y. Rubner, C. Tomasi, and L. J. Guibas. The earth mover’s distance as a metric for image retrieval. International journal of computer vision, 40(2):99–121, 2000. 6 find
  33. A. Suhr, S. Zhou, I. Zhang, H. Bai, and Y. Artzi. A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:1811.00491, 2018. 2 find our card
  34. D. Teney, P. Anderson, X. He, and A. van den Hengel. Tips and tricks for visual question answering: Learnings from the 2017 challenge. arXiv preprint arXiv:1708.02711, 2017. 2, 15 find
  35. D. Teney, L. Liu, and A. van den Hengel. Graph-structured representations for visual question answering. arXiv preprint, 2017. 2 find
  36. B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. Yfcc100m: The new data in multimedia research. arXiv preprint arXiv:1503.01817, 2015. 4, 17 find
  37. A. Torralba and A. A. Efros. Unbiased look at dataset bias. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1521–1528. IEEE, 2011. 2 find
  38. Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 21–29, 2016. 1 find
  39. P. Zhang, Y. Goyal, D. Summers-Stay, D. Batra, and D. Parikh. Yin and yang: Balancing and answering binary visual questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5014–5022, 2016. 1, 2 find
  40. S. Zhang, L. Qu, S. You, Z. Yang, and J. Zhang. Automatic generation of grounded visual questions. arXiv preprint arXiv:1612.06530, 2016. 3 — [arXiv:1612.06530](https://arxiv.org/abs/1612.06530) find
  41. Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. Visual7W: Grounded question answering in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4995–5004, 2016. 1, 2, 3 find our card