Elamur Research
☰
home / papers / 2016 / Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1602.07332

Summary

Image classification and object detection have advanced quickly, but tasks that require reasoning about a scene, such as describing it or answering questions about it, still lag. The authors argue that one reason is that most training data was built for recognition and labels isolated objects, not how those objects interact. To address this, they present Visual Genome, a crowdsourced dataset of densely annotated images. Each image has many localized region descriptions, objects with bounding boxes, attributes, pairwise relationships and question–answer pairs. Region-level annotations are turned into small graphs and then combined into one scene graph per image. Objects, attributes and relationships are mapped to WordNet synsets, so the same concept can be queried consistently across the collection. The authors call the result the densest and largest resource of its type. They report baseline experiments on attribute and relationship classification, region captioning and open-ended QA. These experiments show the dataset is useful for training and also reveal a hard, long-tailed answer space. The resource targets dense captioning, visual QA, visual relationship extraction and semantic image retrieval.

Key points

  • The first release contains 108,249 images taken from the overlap of YFCC100M and MS-COCO.
  • Each image has on average about 21 objects, 18 attributes and 18 relationships, more than 42 region descriptions, and about 17 QA pairs (both freeform and region-based).
  • The dataset is organized into seven parts: descriptions of individual regions, objects together with their attributes and the relationships between them, graphs at both the region and scene level, and QAs. All concepts are canonicalized to WordNet synset IDs.
  • In region description generation, NeuralTalk trained on Visual Genome far outperforms the same model trained on Flickr8K (CIDEr 0.30 vs 0.05; human column 43.03% vs 1.6%).
  • QA answers are strongly long-tailed. The top 100/500/1000 answers cover only 41.1%/57.3%/64.1% of correct answers, compared with 63%/75%/80% in VQA. "Where" and "why" questions are the hardest.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

669 papers of our library cite it (of the 140673 whose bibliographies we read); 51 of them have a page here, 618 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1602.07332 · arXiv 1602.07332

@misc{krishna2016visual,
  title = {{Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations}},
  author = {Krishna, Ranjay and Zhu, Yuke and Groth, Oliver and Johnson, Justin and Hata, Kenji and Kravitz, Joshua and Chen, Stephanie and Kalantidis, Yannis and Li, Li-Jia and Shamma, David A. and Bernstein, Michael S. and Li, Fei-Fei},
  year = {2016},
  doi = {10.48550/arxiv.1602.07332},
  eprint = {1602.07332},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1602.07332}
}

References

122 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Antol et al., 2015. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. (2015). Vqa: Visual question answering. arXiv preprint arXiv:1505.00468. — [arXiv:1505.00468](https://arxiv.org/abs/1505.00468) find
  2. Baker et al., 1998. Baker, C. F., Fillmore, C. J., and Lowe, J. B. (1998). The berkeley framenet project. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics - Volume 1, ACL ’98, pages 86– 90, Strou… find our card
  3. Betteridge, J., Carlson, A., Hong, S. A., Hruschka Jr, E. R., Law, E. L., Mitchell, T. M., and Wang, S. H. (2009). Toward never ending language learning. In AAAI Spring Symposium: Learning by Reading and Learning to Read, pages 1–2. find
  4. Bruner, 1990. Bruner, J. (1990). Culture and human development: A new look. Human development, 33(6):344–355. find
  5. Bunescu and Mooney, 2005. find
  6. Bunescu, R. C. and Mooney, R. J. (2005). A shortest path dependency kernel for relation extraction. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 724–731. Association for Computational Linguistics. find
  7. Chang et al., 2014. Chang, A. X., Savva, M., and Manning, C. D. (2014). Semantic parsing for text to 3d scene generation. find
  8. Chen et al., 2015. Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., and Zitnick, C. L. (2015). Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. find our card
  9. Chen et al., 2014. Chen, X., Liu, Z., and Sun, M. (2014). A unified model for word sense representation and disambiguation. In EMNLP, pages 1025–1035. Citeseer. find
  10. Chen et al., 2013. Chen, X., Shrivastava, A., and Gupta, A. (2013). Neil: Extracting visual knowledge from web data. In Computer Vision (ICCV), 2013 IEEE International Conference on, pages 1409–1416. IEEE. find
  11. Choi et al., 2013. Choi, W., Chao, Y.-W., Pantofaru, C., and Savarese, S. (2013). Understanding indoor scenes using 3d geometric phrases. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 33–40. IEEE. find
  12. Culotta and Sorensen, 2004. find
  13. Culotta, A. and Sorensen, J. (2004). Dependency tree kernels for relation extraction. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, page 423. Association for Computational Linguistics. find
  14. Dauphin, Y. N., de Vries, H., Chung, J., and Bengio, Y. (2015). Rmsprop and equilibrated adaptive learning rates for non-convex optimization. arXiv preprint arXiv:1502.04390. — [arXiv:1502.04390](https://arxiv.org/abs/1502.04390) find
  15. Deng et al., 2009. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. find
  16. CVPR 2009. IEEE Conference on, pages 248–255. IEEE. find
  17. Dollar, P., Wojek, C., Schiele, B., and Perona, P. (2012). Pedestrian detection: An evaluation of the state of the art. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34(4):743–761. find
  18. Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. (2010). The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303–338. find
  19. Farhadi, A., Endres, I., Hoiem, D., and Forsyth, D. (2009). Describing objects by their attributes. In Computer Vision and Pattern Recognition, 2009. find
  20. CVPR 2009. IEEE Conference on, pages 1778–1785. IEEE. find
  21. Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., and Forsyth, D. (2010). Every picture tells a story: Generating sentences from images. In Computer Vision–ECCV 2010, pages 15–29. Springer. find
  22. Fei-Fei, L., Fergus, R., and Perona, P. (2007). Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Image Understanding, 106(1):59–70. find
  23. Ferrari and Zisserman, 2007. find
  24. Ferrari, V. and Zisserman, A. (2007). Learning visual attributes. In Advances in Neural Information Processing Systems, pages 433–440. find
  25. Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A. A., Lally, A., Murdock, J. W., Nyberg, E., Prager, J., et al. (2010). Building watson: An overview of the deepqa project. AI magazine, 31(3):59–79. find
  26. Firestone and Scholl, 2015. find
  27. Firestone, C. and Scholl, B. J. (2015). Cognition does not affect perception: Evaluating the evidence for top-down effects. Behavioral and brain sciences, pages 1–72. find
  28. Forbus, 1984. Forbus, K. D. (1984). Qualitative process theory. Artificial intelligence, 24(1):85–168. find
  29. Gao et al., 2015. Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., and Xu, W. (2015). Are you talking to a machine? dataset and methods for multilingual image question answering. arXiv preprint arXiv:1505.05612. — [arXiv:1505.05612](https://arxiv.org/abs/1505.05612) find
  30. Geman et al., 2015. Geman, D., Geman, S., Hallonquist, N., and Younes, L. (2015). Visual turing test for computer vision systems. Proceedings of the National Academy of Sciences, 112(12):3618–3623. find
  31. Girshick, 2015. Girshick, R. (2015). Fast r-cnn. arXiv preprint arXiv:1504.08083. — [arXiv:1504.08083](https://arxiv.org/abs/1504.08083) find
  32. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 580–587. IEEE. find our card
  33. Goering, C., Rodner, E., Freytag, A., and Denzler, J. (2014). Nonparametric part transfer for fine-grained recognition. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 2489–2496. IEEE. find
  34. Griffin, G., Holub, A., and Perona, P. (2007). Caltech-256 object category dataset. find
  35. GuoDong, Z., Jian, S., Jie, Z., and Min, Z. (2005). Exploring various knowledge in relation extraction. In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 427–434. Association for Computational Linguistics. find
  36. Gupta, A. and Davis, L. S. (2008). Beyond nouns: Exploiting prepositions and comparative adjectives for learning visual classifiers. In Computer Vision– ECCV 2008, pages 16–29. Springer. find
  37. Gupta et al., 2009. Gupta, A., Kembhavi, A., and Davis, L. S. (2009). Observing human-object interactions: Using spatial and functional compatibility for recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 31(10):1775–1789. find
  38. Hayes, 1978. Hayes, P. J. (1978). The naive physics manifesto. Institut pour les études sémantiques et cognitives/Université de Genève. find
  39. Hayes, 1985. Hayes, P. J. (1985). The second naive physics manifesto. Theories of the Commonsense World, pages 1– 36. find
  40. Hearst, M. A., Dumais, S. T., Osman, E., Platt, J., and Scholkopf, B. (1998). Support vector machines. Intelligent Systems and their Applications, IEEE, 13(4):18–28. find
  41. Hochreiter and Schmidhuber, 1997. find
  42. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8):1735–1780. find
  43. Hodosh, M., Young, P., and Hockenmaier, J. (2013). Framing image description as a ranking task: Data, models and evaluation metrics. J. Artif. Int. Res., 47(1):853–899. find
  44. J., Noy, N. F., and Musen, M. A. (2002). A template-based approach toward acquisition of logical sentences. In Intelligent Information Processing, pages 77–89. Springer. find
  45. Huang et al., 2008. Huang, G. B., Mattar, M., Berg, T., and Learned-Miller, E. (2008). Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on Faces in’Real-Life’Images: Detection, Alignment, and Recognition. find
  46. Isola et al., 2015. Isola, P., Lim, J. J., and Adelson, E. H. (2015). Discovering states and transformations in image collections. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1383–1391. find
  47. Izadinia, H., Sadeghi, F., and Farhadi, A. (2014). Incorporating scene context and object layout into appearance modeling. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 232–239. IEEE. find
  48. Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D. A., Bernstein, M., and Fei-Fei, L. (2015). Image retrieval using scene graphs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  49. Karpathy and Fei-Fei, 2014. find
  50. Karpathy, A. and Fei-Fei, L. (2014). Deep visual-semantic alignments for generating image descriptions. arXiv preprint arXiv:1412.2306. find our card
  51. Kiros et al., 2014. Kiros, R., Salakhutdinov, R., and Zemel, R. (2014). Multimodal neural language models. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 595–603. find
  52. Krishna, R., Hata, K., Chen, S., Kravitz, J., Shamma, D. A., Fei-Fei, L., and Bernstein, M. S. (2016). Embracing error to enable rapid crowdsourcing. In CHI’16-SIGCHI Conference on Human Factors in Computing System. find
  53. Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105. find
  54. Lampert, C. H., Nickisch, H., and Harmeling, S. (2009). Learning to detect unseen object classes by between-class attribute transfer. In Computer Vision and Pattern Recognition, 2009. find
  55. CVPR 2009. IEEE Conference on, pages 951–958. IEEE. find
  56. Leacock, C., Miller, G. A., and Chodorow, M. (1998). Using corpus statistics and wordnet relations for sense identification. Computational Linguistics, 24(1):147–165. find
  57. Lebret, R., Pinheiro, P. O., and Collobert, R. (2015). Phrase-based image captioning. arXiv preprint arXiv:1502.03671. — [arXiv:1502.03671](https://arxiv.org/abs/1502.03671) find
  58. Lin et al., 2014. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014, pages 740–755. Springer. find our card
  59. Lu et al., 2016. Lu, C., Krishna, R., Bernstein, M. S., and Fei-Fei, L. (2016). Visual relationship detection using language priors. find
  60. Ma et al., 2015. Ma, L., Lu, Z., and Li, H. (2015). Learning to answer questions from image using convolutional neural network. arXiv preprint arXiv:1506.00333. — [arXiv:1506.00333](https://arxiv.org/abs/1506.00333) find
  61. Malinowski and Fritz, 2014. find
  62. Malinowski, M. and Fritz, M. (2014). A multi-world approach to question answering about real-world scenes based on uncertain input. In Advances in Neural Information Processing Systems, pages 1682–1690. find
  63. Malinowski, M., Rohrbach, M., and Fritz, M. (2015). Ask your neurons: A neural-based approach to answering questions about images. arXiv preprint arXiv:1505.01121. — [arXiv:1505.01121](https://arxiv.org/abs/1505.01121) find
  64. Malisiewicz et al., 2008. find
  65. Malisiewicz, T., Efros, A., et al. (2008). Recognition by association via learning per-exemplar distances. In Computer Vision and Pattern Recognition, 2008. find
  66. CVPR 2008. IEEE Conference on, pages 1–8. IEEE. find
  67. Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. J., and McClosky, D. (2014). The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. find our card
  68. Mao et al., 2014. Mao, J., Xu, W., Yang, Y., Wang, J., and Yuille, A. L. (2014). Explain images with multimodal recurrent neural networks. arXiv preprint arXiv:1410.1090. — [arXiv:1410.1090](https://arxiv.org/abs/1410.1090) find
  69. Mihalcea, R., Chklovski, T. A., and Kilgarriff, A. (2004). The senseval-3 english lexical sample task. Association for Computational Linguistics. find
  70. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  71. Miller, 1995. Miller, G. A. (1995). Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41. find our card
  72. Nathan Silberman and Fergus, 2012. find
  73. Nathan Silberman, Derek Hoiem, P. K. and Fergus, R. (2012). Indoor segmentation and support inference from rgbd images. In ECCV. find
  74. Niu et al., 2012. Niu, F., Zhang, C., Ré, C., and Shavlik, J. (2012). Elementary: Large-scale knowledge-base construction via machine learning and statistical inference. International Journal on Semantic Web and Information Systems (IJSWIS), 8(3):42–73. find
  75. Ordonez, V., Kulkarni, G., and Berg, T. L. (2011). Im2text: Describing images using 1 million captioned photographs. In Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., and Weinberger, K., editors, Advances in Neural Information Processing Systems 24, pages 1143–1151. Curran Associates, Inc. find
  76. Pal and Saha, 2015. Pal, A. R. and Saha, D. (2015). Word sense disambiguation: a survey. arXiv preprint arXiv:1508.01346. — [arXiv:1508.01346](https://arxiv.org/abs/1508.01346) find
  77. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
  78. Patterson, G., Xu, C., Su, H., and Hays, J. (2014). The sun attribute database: Beyond categories for deeper scene understanding. International Journal of Computer Vision, 108(1-2):59–81. find
  79. Perronnin et al., 2010. Perronnin, F., Sánchez, J., and Mensink, T. (2010). Improving the fisher kernel for large-scale image classification. In Computer Vision–ECCV 2010, pages 143–156. Springer. find
  80. Prest et al., 2012. Prest, A., Schmid, C., and Ferrari, V. (2012). Weakly supervised learning of interactions between humans and objects. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34(3):601–614. find
  81. Ramanathan, V., Li, C., Deng, J., Han, W., Li, Z., Gu, K., Song, Y., Bengio, S., Rossenberg, C., and Fei-Fei, L. (2015). Learning semantic relationships for better action retrieval in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1100–1109. find
  82. Ren et al., 2015a. Ren, M., Kiros, R., and Zemel, R. (2015a). Image question answering: A visual semantic embedding model and a new dataset. arXiv preprint arXiv:1505.02074. find
  83. Ren et al., 2015b. Ren, S., He, K., Girshick, R., and Sun, J. (2015b). Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01497. find our card
  84. Rothe and Schütze, 2015. Rothe, S. and Schütze, H. (2015). Autoextend: Extending word embeddings to embeddings for synsets and lexemes. arXiv preprint arXiv:1507.01127. find
  85. Ruggero Ronchi and Perona, 2015. find
  86. Ruggero Ronchi, M. and Perona, P. (2015). Describing Common Human Visual Actions in Images. ArXiv e-prints. find
  87. Russakovsky et al., 2015. find
  88. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), pages 1–42. find our card
  89. Russell, B. C., Torralba, A., Murphy, K. P., and Freeman, W. T. (2008). Labelme: a database and web-based tool for image annotation. International journal of computer vision, 77(1-3):157–173. find
  90. Sadeghi, F., Divvala, S. K., and Farhadi, A. (2015). Viske: Visual knowledge extraction and question answering by visual verification of relation phrases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1456–1464. find
  91. Sadeghi and Farhadi, 2011. find
  92. Sadeghi, M. A. and Farhadi, A. (2011). Recognition using visual phrases. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1745–1752. IEEE. find
  93. Salehi, N., Irani, L. C., and Bernstein, M. S. (2015). We are dynamo: Overcoming stalling and friction in collective action for crowd workers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 1621–1630. ACM. find
  94. Schank and Abelson, 2013. find
  95. Schank, R. C. and Abelson, R. P. (2013). Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Psychology Press. find
  96. Schuler, 2005. Schuler, K. K. (2005). Verbnet: A Broad-coverage, Comprehensive Verb Lexicon. PhD thesis, Philadelphia, PA, USA. AAI3179808. find
  97. Schuster, S., Krishna, R., Chang, A., Fei-Fei, L., and Manning, C. D. (2015). Generating semantically precise scene graphs from textual descriptions for improved image retrieval. find
  98. Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y. (2013). Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229. find
  99. Simonyan and Zisserman, 2014. find
  100. Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. find our card
  101. Snow et al., 2008. Snow, R., O’Connor, B., Jurafsky, D., and Ng, A. Y. (2008). Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the conference on empirical methods in natural language processing, pages 254–263. Association for Computatio… find
  102. Socher, R., Huval, B., Manning, C. D., and Ng, A. Y. (2012). Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211. Association for… find
  103. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2014). Going deeper with convolutions. arXiv preprint arXiv:1409.4842. — [arXiv:1409.4842](https://arxiv.org/abs/1409.4842) find
  104. Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J. (2016). Yfcc100m: The new data in multimedia research. Commun. ACM, 59(2):64–73. find
  105. Torralba, A., Fergus, R., and Freeman, W. T. (2008). 80 million tiny images: A large data set for nonparametric object and scene recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 30(11):1958–1970. find
  106. Varma and Zisserman, 2005. find
  107. Varma, M. and Zisserman, A. (2005). A statistical approach to texture classification from single images. International Journal of Computer Vision, 62(1-2):61–81. find
  108. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2014). Show and tell: A neural image caption generator. arXiv preprint arXiv:1411.4555. find our card
  109. Wah et al., 2011. Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. (2011). The caltech-ucsd birds-200-2011 dataset. find
  110. Xiao et al., 2010. Xiao, J., Hays, J., Ehinger, K., Oliva, A., Torralba, A., et al. (2010). Sun database: Large-scale scene recognition from abbey to zoo. In Computer vision and pattern recognition (CVPR), 2010 IEEE conference on, pages 3485–3492. IEEE. find
  111. Xu et al., 2015. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A. C., Salakhutdinov, R., Zemel, R. S., and Bengio, Y. (2015). Show, attend and tell: Neural image caption generation with visual attention. CoRR, abs/1502.03044. find our card
  112. Yao, B. and Fei-Fei, L. (2010). Modeling mutual context of object and human pose in human-object interaction activities. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 17–24. IEEE. find
  113. Yao et al., 2007. Yao, B., Yang, X., and Zhu, S.-C. (2007). Introduction to a large-scale general purpose ground truth database: methodology, annotation tool and benchmarks. In Energy Minimization Methods in Computer Vision and Pattern Recognition, pages 169–183. Springer. find
  114. Young et al., 2014. Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. find
  115. Yu et al., 2015. Yu, L., Park, E., Berg, A. C., and Berg, T. L. (2015). Visual Madlibs: Fill in the blank Image Generation and Question Answering. arXiv preprint arXiv:1506.00278. — [arXiv:1506.00278](https://arxiv.org/abs/1506.00278) find
  116. Zeng et al., 2014. Zeng, D., Liu, K., Lai, S., Zhou, G., and Zhao, J. (2014). Relation classification via convolutional deep neural network. In Proceedings of COLING, pages 2335–2344. find
  117. Zhou et al., 2007. Zhou, G., Zhang, M., Ji, D. H., and Zhu, Q. (2007). Tree kernel-based relation extraction with context-sensitive structured parse tree information. find
  118. Zhu et al., 2009. Zhu, J., Nie, Z., Liu, X., Zhang, B., and Wen, J.-R. (2009). Statsnowball: a statistical approach to extracting entity relationships. In Proceedings of the 18th international conference on World wide web, pages 101– 110. ACM. find
  119. Zhu et al., 2014. Zhu, Y., Fathi, A., and Fei-Fei, L. (2014). Reasoning about Object Affordances in a Knowledge Base Representation. In European Conference on Computer Vision. find
  120. Zhu et al., 2015. Zhu, Y., Zhang, C., Ré, C., and Fei-Fei, L. (2015). Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries. In arXiv preprint arXiv:1507.05670. — [arXiv:1507.05670](https://arxiv.org/abs/1507.05670) find
  121. Zitnick and Parikh, 2013. find
  122. Zitnick, C. L. and Parikh, D. (2013). Bringing semantics into focus using visual abstraction. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 3009–3016. IEEE. find