Elamur Research⌕
☰
home / papers / 2015 / Exploring Models and Data for Image Question Answering

Exploring Models and Data for Image Question Answering

Paper · 2015 · Mengye Ren, Ryan Kiros and Richard Zemel

Summary

The paper addresses image question answering: given a picture and a short natural-language question, the system must produce a correct answer. The authors restrict answers to a single word, which turns the task into classification and makes evaluation simpler and more reliable. Their main model, VIS+LSTM, takes CNN features from a frozen 19-layer VGG network, projects them linearly into the word-embedding space, and feeds them to an LSTM as though the image were an extra word in the question. A softmax over candidate answers follows the LSTM. Variants add a backward LSTM or place the image at both ends of the question. They compare this model against simpler baselines such as IMG+BOW and question-only ("blind") models. Because the existing DAQUAR benchmark is small and skewed, they also built a parser-based algorithm that rewrites MS-COCO captions into questions. The result is COCO-QA, a much larger dataset with a more balanced answer distribution. Their models beat prior published DAQUAR results by roughly 1.8×. Notably, a plain bag-of-words question encoder performed about as well as the recurrent one.

Key points

  • DAQUAR has about 1,500 indoor images and 7,000 questions. It comes in 37-class and 894-class variants, and guessing the most frequent answer already scores well on it.
  • COCO-QA covers four question types: object, number, color and location. Its training split has 78,736 questions, of which 54,992 are object questions, 5,885 number, 13,059 color and 4,800 location.
  • The question generator works as follows:
    • It parses captions with the Stanford parser and splits compound sentences.
    • It swaps "a(n)" for "the" and applies wh-movement under simple constraints such as A-over-A.
    • It uses WordNet and NLTK to identify candidate answers.
  • Filtering out answers that were too rare or too frequent reduced the share of the most common answer in the COCO-QA test set from 24.98% to 7.30%.
  • The full model improved accuracy by about 50% over the image-only baseline. The models handled color questions well, but counting remained weak and failed when several object types appeared in one image.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TorontoToronto · Canada110 works · 179,062 citations🔬Canadian Institute for Advanced ResearchToronto · Canada35 works · 19,316 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 487 works

Cite

Cite as Mengye Ren, Ryan Kiros and Richard Zemel. 2015. Exploring Models and Data for Image Question Answering. arXiv:1505.02074.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1505.02074 · arXiv 1505.02074

@misc{ren2015exploring,
  title = {{Exploring Models and Data for Image Question Answering}},
  author = {Mengye Ren, Ryan Kiros and Richard Zemel},
  year = {2015},
  doi = {10.48550/arxiv.1505.02074},
  eprint = {1505.02074},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1505.02074}
}

References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in CVPR, 2015.
  2. R. Kiros, R. Salakhutdinov, and R. S. Zemel, “Unifying visual-semantic embeddings with multimodal neural language models,” TACL, 2015. COCOQA 14855 Where are the ripe bananas sitting? Ground truth: basket IMG+BOW: basket (0.97) 2-VIS+BLSTM: basket (0.58) BOW: bowl (0.48) DAQUAR 585 What is the objec…
  3. A. Karpathy, A. Joulin, and L. Fei-Fei, “Deep fragment embeddings for bidirectional image sentence mapping,” in NIPS, 2013. find
  4. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille, “Explain images with multimodal recurrent neural networks,” NIPS Deep Learning Workshop, 2014. find
  5. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” in CVPR, 2014.
  6. X. Chen and C. L. Zitnick, “Learning a recurrent visual representation for image caption generation,” CoRR, vol. abs/1411.5654, 2014. find
  7. H. Fang, S. Gupta, F. N. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt, C. L. Zitnick, and G. Zweig, “From captions to visual concepts and back,” in CVPR, 2015.
  8. K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in ICML, 2015.
  9. R. Lebret, P. O. Pinheiro, and R. Collobert, “Phrase-based image captioning,” in ICML, 2015. find
  10. B. Klein, G. Lev, G. Lev, and L. Wolf, “Fisher vectors derived from hybrid Gaussian-Laplacian mixture models for image annotations,” in CVPR, 2015. find
  11. M. Malinowski and M. Fritz, “Towards a visual Turing challenge,” in NIPS Workshop on Learning Semantics, 2014. find
  12. N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from RGBD images,” in ECCV, 2012. find
  13. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “VQA: Visual Question Answering,” CoRR, vol. abs/1505.00468, 2015. find
  14. M. Malinowski, M. Rohrbach, and M. Fritz, “Ask Your Neurons: A Neural-based Approach to Answering Questions about Images,” CoRR, vol. abs/1505.01121, 2015. — [arXiv:1505.01121](https://arxiv.org/abs/1505.01121) find
  15. H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu, “Are you talking to a machine? dataset and methods for multilingual image question answering,” CoRR, vol. abs/1505.05612, 2015. find
  16. L. Ma, Z. Lu, and H. Li, “Learning to answer questions from image using convolutional neural network,” CoRR, vol. abs/1506.00333, 2015. find
  17. T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common Objects in Context,” in ECCV, 2014.
  18. X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick, “Microsoft COCO captions: Data collection and evaluation server,” CoRR, vol. abs/1504.00325, 2015.
  19. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. find
  20. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in ICLR, 2015.
  21. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” IJCV, 2015.
  22. T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” in ICLR, 2013.
  23. A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov, “DeViSE: A deep visual-semantic embedding model,” in NIPS, 2013. find
  24. M. Hodosh, P. Young, and J. Hockenmaier, “Framing image description as a ranking task: Data, models and evaluation metrics,” J. Artif. Intell. Res. (JAIR), vol. 47, pp. 853–899, 2013. find
  25. V. Ordonez, G. Kulkarni, and T. L. Berg, “Im2text: Describing images using 1 million captioned photographs,” in NIPS, 2011. find
  26. D. Klein and C. D. Manning, “Accurate unlexicalized parsing,” in ACL, 2003.
  27. N. Chomsky, Conditions on Transformations. New York: Academic Press, 1973. find
  28. C. Fellbaum, Ed., WordNet An Electronic Lexical Database. Cambridge, MA; London: The MIT Press, May 1998. find
  29. S. Bird, “NLTK: the natural language toolkit,” in ACL, 2006.
  30. J. Devlin, S. Gupta, R. Girshick, M. Mitchell, and C. L. Zitnick, “Exploring nearest neighbor approaches for image captioning,” CoRR, vol. abs/1505.04467, 2015. find
  31. Z. Wu and M. Palmer, “Verb semantics and lexical selection,” in ACL, 1994.
  32. M. Malinowski and M. Fritz, “A multi-world approach to question answering about real-world scenes based on uncertain input,” in NIPS, 2014. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ren-2015-exploring-models-data-image-question/