Elamur Research
☰
home / papers / 2016 / CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1612.06890

Summary

Visual question answering (VQA) benchmarks are meant to measure whether systems can reason about images. In practice, models can score well on them by exploiting statistical regularities in the questions, and a single wrong answer can come from several different failures. CLEVR is a synthetic diagnostic benchmark built to separate these causes. Its images are Blender renderings of scenes with 3 to 10 simple 3D objects. The questions are generated automatically, and each one is stored as a functional program that can be run on the scene graph to get the ground-truth answer. Rejection sampling within families of related questions limits answer bias, and degenerate shortcut questions are filtered out. Because no outside knowledge is needed, accuracy can be broken down by question type, length and structure. The authors used CLEVR to test several modern VQA models. They found that the models struggle on tasks that need short-term memory, such as comparing attributes, and on compositional generalization, such as recognizing unseen attribute combinations. They present CLEVR as a complement to real-world VQA data for measuring reasoning, not as a goal in itself.

Key points

  • Size: 100k rendered images and about 1M generated questions, of which 853,554 are unique. For comparison, SHAPES has 15,616 questions and only 244 unique ones.
  • Object world: 3 shapes (cube, sphere, cylinder), 2 sizes, 2 materials (metal, rubber) and 8 colors. There are 4 spatial relations defined geometrically from the camera viewpoint, plus a "same-attribute" relation.
  • Question programs: each question is a chain- or tree-structured composition of basic functions such as filter, relate, unique, count, exist, query, compare and and/or.
  • Annotations: every image comes with a full scene graph of object positions and attributes, so the vision step can be replaced by "perfect sight."
  • Availability: the dataset and the code to generate new images and questions were announced for public release.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

352 papers of our library cite it (of the 140673 whose bibliographies we read); 26 of them have a page here, 326 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1612.06890 · arXiv 1612.06890

@misc{johnson2016clevr,
  title = {{CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning}},
  author = {Johnson, Justin and Hariharan, Bharath and Maaten, Laurens van der and Fei-Fei, Li and Zitnick, C. Lawrence and Girshick, Ross},
  year = {2016},
  doi = {10.48550/arxiv.1612.06890},
  eprint = {1612.06890},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1612.06890}
}

References

50 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Agrawal, D. Batra, and D. Parikh. Analyzing the behavior of visual question answering models. In EMNLP, 2016. 1 find
  2. J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Learning to compose neural networks for question answering. In NAACL, 2016. 1, 2, 4, 5, 8, 11 find our card
  3. J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks. In CVPR, 2016. 1, 2, 4, 8, 11 find our card
  4. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Zitnick, and D. Parikh. VQA: Visual question answering. In ICCV, 2015. 1, 2, 4, 5, 11 find
  5. Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. TPAMI, 35(8):1798–1828, 2014. 8 find our card
  6. Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Blender Institute, Amsterdam, 2016. 3 find
  7. J. Chen, P. Kuznetsova, D. Warren, and Y. Choi. Deja image-captions: A corpus of expressive image descriptions in repetition. In NAACL, 2015. 2 find
  8. A. Farhadi, M. Hejrati, A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences for images. In ECCV, 2010. 2 find
  9. A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In arXiv:1606.01847, 2016. 1, 4, 5 find
  10. H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu. Are you talking to a machine? Dataset and methods for multilingual image question answering. In NIPS, 2015. 1, 2, 4 find
  11. Y. Gao, O. Beijbom, N. Zhang, and T. Darrell. Compact bilinear pooling. In CVPR, 2016. 5 find
  12. D. Geman, S. Geman, N. Hallonquist, and L. Younes. Visual Turing test for computer vision systems. Proceedings of the National Academy of Sciences, 112(12):3618–3623, 2015. 2 find
  13. A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Barwinska, S. Colmenarejo, E. Grefenstette, T. Ramalho, J. Agapiou, A. Badia, K. Hermann, Y. Zwols, G. Ostrovski, A. Cain, H. King, C. Summerfield, P. Blunsom, K. . Kavukcuoglu, and D. Hassabis. Hybrid computing using a neural net… find
  14. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016. 5 find
  15. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997. 2, 5 find
  16. A. Jabri, A. Joulin, and L. van der Maaten. Revisiting visual question answering baselines. In ECCV, 2016. 1, 4 find
  17. J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei. Image retrieval using scene graphs. In CVPR, 2015. 3 find
  18. A. Joulin and T. Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In NIPS, 2015. 8 find
  19. S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014. 2 find
  20. D. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 5 find
  21. R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L. Jia-Li, D. Shamma, M. Bernstein, and L. Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2016. 1, 2, 3 find our card
  22. H. J. Levesque, E. Davis, and L. Morgenstern. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, page 47, 2011. 2 find
  23. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 2 find our card
  24. J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical question-image co-attention for visual question answering. In NIPS, 2016. 1, 2, 4 find
  25. L. Ma, Z. Lu, and H. Li. Learning to answer questions from image using convolutional neural network. In AAAI, 2016. 4 find
  26. M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, 2014. 1, 2 find
  27. M. Malinowski and M. Fritz. Towards a visual Turing challenge. In NIPS 2014 Workshop on Learning Semantics, 2014. 2 find
  28. M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions about images. In ICCV, 2015. 2, 4 find
  29. T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. In arXiv 1301.3781, 2013. 5 find
  30. O. Pfungst. Clever Hans (The horse of Mr. von Osten): A contribution to experimental animal and human psychology. Henry Holt, New York, 1911. 1 find
  31. A. Ray, G. Christie, M. Bansal, D. Batra, and D. Parikh. Question relevance in vqa: Identifying non-visual and false-premise questions. In EMNLP, 2016. 1 find
  32. M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. In NIPS, 2015. 1, 2, 4 find
  33. K. Shih, S. Singh, and D. Hoiem. Where to look: Focus regions for visual question answering. In CVPR, 2016. 4 find
  34. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929–1958, 2014. 5 find
  35. B. Sturm. A simple method to determine if a music information retrieval system is a horse. IEEE Transactions on Multimedia, 16(6):1636–1644, 2014. 1 find
  36. B. Sturm. Horse taxonomy and taxidermy. HORSE2016, 2016. 1 find
  37. M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler. Movieqa: Understanding stories in movies through question-answering. In CVPR, 2016. 2 find
  38. J. Weston, A. Bordes, S. Chopra, A. Rush, B. van Merriënboer, A. Joulin, and T. Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. In ICLR, 2016. 2 find our card
  39. J. Weston, S. Chopra, and A. Bordes. Memory networks. In ICLR, 2015. 8 find
  40. T. Winograd. Understanding Natural Language. Academic Press, 1972. 2 find
  41. Q. Wu, C. Shen, A. van den Hengel, P. Wang, and A. Dick. Image captioning and visual question answering based on attributes and their related external knowledge. In arXiv 1603.02814, 2016. 4 — [arXiv:1603.02814](https://arxiv.org/abs/1603.02814) find
  42. C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering. ICML, 2016. 4, 8 find
  43. H. Xu and K. Saenko. Ask, attend, and answer: Exploring question-guided spatial attention for visual question answering. In ECCV, 2016. 4 find
  44. Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering. In CVPR, 2016. 1, 2, 4, 5, 8 find
  45. P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. In TACL, pages 67–78, 2014. 2 find
  46. L. Yu, E. Park, A. Berg, and T. Berg. Visual madlibs: Fill in the blank image generation and question answering. In ICCV, 2015. 1, 2 find
  47. P. Zhang, Y. Goyal, D. Summers-Stay, D. Batra, and D. Parikh. Yin and yang: Balancing and answering binary visual questions. In CVPR, 2016. 1, 2 find
  48. B. Zhou, Y. Tian, S. Sukhbataar, A. Szlam, and R. Fergus. Simple baseline for visual question answering. In arXiv:1512.02167, 2015. 1, 4, 5 — [arXiv:1512.02167](https://arxiv.org/abs/1512.02167) find
  49. Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. Visual7w: Grounded question answering in images. In CVPR, 2016. 1, 2, 4 find our card
  50. C. Zitnick and D. Parikh. Bringing semantics into focus using visual abstraction. In CVPR, 2013. 2 find