Elamur Research
☰
home / papers / 2015 / Generation and Comprehension of Unambiguous Object Descriptions

Generation and Comprehension of Unambiguous Object Descriptions

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1511.02283

Summary

The paper addresses referring expressions: natural-language phrases that identify exactly one object or region in an image. It covers two tasks. In generation, the system writes a phrase that singles out a highlighted object. In comprehension, it picks the matching bounding box from a set of candidates, given a phrase. Unlike open-ended image captioning, these tasks can be scored objectively, since a description is good if a listener can locate the intended object from it. The authors built Google Refexp (G-Ref), a large dataset on top of MS-COCO, through iterative Mechanical Turk annotation in which one worker wrote each description and another checked it. Their model combines CNN region features with an LSTM language model. A baseline trains it like a standard captioner. The full model adds discriminative, margin-based training that penalises descriptions which also fit other regions in the same image. The full model beat the baseline on both tasks and produced longer, less ambiguous phrases. It also supports semi-supervised training on automatically generated descriptions. The work shows that modelling the listener improves grounded language generation.

Key points

  • G-Ref dataset: 104,560 verified expressions for 54,822 objects in 26,711 MS-COCO images, about 1.91 expressions per object. Objects were chosen only if 2–4 instances of the same category appear in the image and the bounding box covers at least 5% of the image. The dataset and an evaluation toolbox are released at github.com/mjhucla/Google_Refexp_toolbox.
  • Comparison with earlier data: ReferIt (Kazemzadeh et al.) has 130,525 expressions over 19,894 ImageCLEF photos. Because its images often contain only one instance of a class and focus on background "stuff", its descriptions tend to be short. The authors also compare G-Ref with the UNC-Ref-COCO dataset.
  • Comprehension, random image split (multibox proposals, ground-truth descriptions): Precision@1 rises from 0.404 to 0.444 on validation and from 0.407 to 0.451 on test, baseline vs. full model. These figures come from a footnote; the main results table (Table 2) was in the omitted middle of the text.
  • Behaviour of the comprehension model: it reacts correctly to single-word changes, e.g. "in black" vs. "in red". It also uses word order to tell which noun is the referent ("dark horse carrying a woman" vs. "woman on the dark horse"). It fails when the described object is absent, or when the object is very small and has few training examples.
  • Uses: natural-language interfaces for robots and photo editing, and a vision-and-language benchmark with a clear accuracy measure.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

184 papers of our library cite it (of the 140673 whose bibliographies we read); 25 of them have a page here, 159 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.02283 · arXiv 1511.02283

@misc{mao2015generation,
  title = {{Generation and Comprehension of Unambiguous Object Descriptions}},
  author = {Mao, Junhua and Huang, Jonathan and Toshev, Alexander and Camburu, Oana and Yuille, Alan and Murphy, Kevin},
  year = {2015},
  doi = {10.48550/arxiv.1511.02283},
  eprint = {1511.02283},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.02283}
}

References

56 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ms coco captioning challenge. http://mscoco.org/dataset/\#captions-challenge2015. 6 ↗
  2. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. arXiv, 2015. 2 find
  3. L. Bahl, P. Brown, P. V. de Souza, and R. Mercer. Maximum mutual information estimation of hidden Markov model parameters for speech recognition. In ICASSP, volume 11, pages 49–52, Apr. 1986. 5 find
  4. D. P. Barrett, S. A. Bronikowski, H. Yu, and J. M. Siskind. Robot language learning, generation, and comprehension. arXiv preprint arXiv:1508.06161, 2015. 2 — [arXiv:1508.06161](https://arxiv.org/abs/1508.06161) find
  5. X. Chen and C. L. Zitnick. Mind’s eye: A recurrent visual representation for image caption generation. In CVPR, 2015. 1, 2 find
  6. M.-M. Cheng, S. Zheng, W.-Y. Lin, V. Vineet, P. Sturgess, N. Crook, N. J. Mitra, and P. Torr. ImageSpirit: Verbal guided image parsing. ACM Trans. Graphics, 2014. 2 find
  7. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009. 4 find
  8. J. Devlin, S. Gupta, R. Girshick, M. Mitchell, and C. L. Zitnick. Exploring nearest neighbor approaches for image captioning. arXiv preprint arXiv:1505.04467, 2015. 1 — [arXiv:1505.04467](https://arxiv.org/abs/1505.04467) find
  9. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015. 1, 2, 4 find our card
  10. D. Erhan, C. Szegedy, A. Toshev, and D. Anguelov. Scalable object detection using deep neural networks. In CVPR, pages 2155–2162, 2014. 4 find
  11. H. J. Escalante, C. A. Hernandez, J. A. Gonzalez, A. Lopez-Lopez, M. Montes, E. F. Morales, L. E. Sucar, L. Villasenor, and M. Grubinger. The segmented and annotated IAPR TC-12 benchmark. CVIU, 2010. 3 find
  12. H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. Platt, et al. From captions to visual concepts and back. In CVPR, 2015. 1, 2 find
  13. A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences from images. In ECCV, pages 15–29. 2010. 1, 2 find
  14. N. FitzGerald, Y. Artzi, and L. S. Zettlemoyer. Learning distributions over logical forms for referring expression generation. In EMNLP, pages 1914–1925, 2013. 1, 2 find
  15. H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu. Are you talking to a machine? dataset and methods for multilingual image question answering. In NIPS, 2015. 2 find
  16. D. Geman, S. Geman, N. Hallonquist, and L. Younes. Visual turing test for computer vision systems. PANS, 112(12):3618–3623, 2015. 2 find
  17. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. 4 find our card
  18. D. Gkatzia, V. Rieser, P. Bartie, and W. Mackaness. From the virtual to the real world: Referring to objects in Real-World spatial scenes. In EMNLP, 2015. 2 find
  19. D. Golland, P. Liang, and D. Klein. A game-theoretic approach to generating spatial descriptions. In EMNLP, pages 410–419, 2010. 1, 2, 5 find
  20. N. D. Goodman and D. Lassiter. Probabilistic semantics and pragmatics: Uncertainty in language and thought. Handbook of Contemporary Semantic Theory. Wiley-Blackwell, 2014. 2 find
  21. K. Greff, R. K. Srivastava, J. Koutnı́k, B. R. Steunebrink, and J. Schmidhuber. Lstm: A search space odyssey. In ICML, 2015. 4 find
  22. H. P. Grice. Logic and conversation. na, 1970. 2 find
  23. M. Hodosh, P. Young, and J. Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. JAIR, 47:853–899, 2013. 2 find
  24. R. Hu, H. Xu, M. Rohrbach, J. Feng, K. Saenko, and T. Darrell. Natural language object retrieval. CVPR, 2016. 2 find
  25. J. Johnson, A. Karpathy, and L. Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. arXiv preprint arXiv:1511.07571, 2015. 2 — [arXiv:1511.07571](https://arxiv.org/abs/1511.07571) find
  26. A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. arXiv preprint arXiv:1412.2306, 2014. 1, 2 find our card
  27. S. Kazemzadeh, V. Ordonez, M. Matten, and T. L. Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, pages 787–798, 2014. 1, 2, 3 find
  28. R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014. 1, 2 find our card
  29. R. Kiros, R. Zemel, and R. Salakhutdinov. Multimodal neural language models. In ICML, 2014. 4 find
  30. E. Krahmer and M. Theune. Efficient context-sensitive generation of referring expressions. Information sharing: Reference and presupposition in language generation and interpretation, 143:223–263, 2002. 2 find
  31. E. Krahmer and K. van Deemter. Computational generation of referring expressions: A survey. Comp. Linguistics, 38, 2012. 1, 2 find
  32. R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. arXiv preprint arXiv:1602.07332, 2016. 2 find our card
  33. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097–1105, 2012. 4 find
  34. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. Baby talk: Understanding and generating image descriptions. In CVPR, 2011. 2 find
  35. A. Lavie and A. Agarwal. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgements. In Workshop on Statistical Machine Translation, pages 228–231, 2007. 6 find
  36. S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y. Choi. Composing simple image descriptions using web-scale n-grams. In CoNLL, pages 220–228, 2011. 2 find
  37. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 2, 3 find our card
  38. M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, pages 1682–1690, 2014. 2 find
  39. M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions about images. In NIPS, 2015. 2 find
  40. J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). In ICLR, 2015. 1, 2, 4 find
  41. M. Mitchell, K. van Deemter, and E. Reiter. Natural reference to objects in a visual domain. In INLG, pages 95–104, 2010. 1, 2 find
  42. M. Mitchell, K. van Deemter, and E. Reiter. Generating expressions that refer to visible objects. In HLT-NAACL, pages 1174–1184, 2013. 1, 2 find
  43. V. Ordonez, G. Kulkarni, and T. L. Berg. Im2text: Describing images using 1 million captioned photographs. In NIPS, 2011. 2 find
  44. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311–318, 2002. 6 find our card
  45. B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV, 2015. 2 find
  46. M. A. Sadeghi and A. Farhadi. Recognition using visual phrases. In CVPR, 2011. 2 find
  47. A. Sadovnik, Y.-I. Chiu, N. Snavely, S. Edelman, and T. Chen. Image description with a goal: Building efficient discriminating expressions for images. In CVPR, 2012. 2 find
  48. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015. 4 find our card
  49. R. Socher, Q. Le, C. Manning, and A. Ng. Grounded compositional semantics for finding and describing images with sentences. In TACL, 2014. 2 find our card
  50. K. van Deemter, I. van der Sluis, and A. Gatt. Building a semantically transparent corpus for the generation of referring expressions. In INLG, pages 130–132, 2006. 1, 2 find
  51. R. Vedantam, C. Lawrence Zitnick, and D. Parikh. CIDEr: Consensus-based image description evaluation. In CVPR, 2015. 6 find our card
  52. J. Viethen and R. Dale. The use of spatial relations in referring expression generation. In INLG, pages 59–67. Association for Computational Linguistics, 2008. 1, 2 find
  53. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In CVPR, 2015. 1, 2, 4 find our card
  54. T. Winograd. Understanding natural language. Cognitive psychology, 3(1):1–191, 1972. 2 find
  55. K. Xu, J. Ba, R. Kiros, C. A. Cho, Kyunghyun, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015. 1, 2 find our card
  56. Y. Yang, C. L. Teo, H. Daumé III, and Y. Aloimonos. Corpus-guided sentence generation of natural images. In EMNLP, pages 444–454, 2011. 2 find