Elamur Research
☰
home / papers / 2016 / Modeling Context in Referring Expressions

Modeling Context in Referring Expressions

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1608.00272

Summary

The paper addresses generating and comprehending referring expressions: phrases that single out one object in an image so a listener can find it. Earlier deep models, notably Mao et al., used features of the whole image as context. The authors argue that this context is too generic, because ambiguity usually comes from other objects of the same category. They make two changes. First, they add visual comparison features ("visdif") that encode how the target differs from same-category objects in appearance (VGG-fc7 differences), location and size, and feed these into an LSTM. Second, they couple the LSTMs for all same-type objects in an image ("tie"), so the resulting expressions are complementary rather than duplicated. The models are evaluated on RefCOCO, RefCOCO+ and RefCOCOg against Baseline and MMI models. Visual comparison and joint generation both improve automatic scores and human-judged accuracy. They also sharply reduce cases where several objects receive identical descriptions. This matters for human–robot dialogue and grounded language systems, where an unambiguous reference is the whole point.

Key points

  • Data and baselines: The work uses three datasets built on MSCOCO (RefCOCO, RefCOCO+, RefCOCOg), released with a toolbox at github.com/lichengunc/refer. Baseline and MMI models are reimplemented following Mao et al.
  • Features and automatic metrics: The models use VGG-fc7 features and a 5-d location/size vector. "visdif+tie" reaches the highest Bleu/Rouge/Meteor in almost every setting. MMI variants score lower on these automatic metrics.
  • Human evaluation: Three Turkers clicked on the object each generated expression describes. On RefCOCO, the share of expressions resolved to the correct object rose from 62.42%/64.99% (Baseline, Test A/B) to 71.40%/76.14% (visdif+tie).
  • Duplicate descriptions: The fraction of RefCOCO images with identical expressions for multiple objects fell from 15.60%/16.40% to 4.27%/5.33%. On RefCOCO+, "visdif+MMI+tie" reduced it from 28.67%/46.27% to 10.13%/13.33%.
  • RefCOCOg limitation: Tied generation was not applied to RefCOCOg. In that dataset, objects from the same image can fall into different train/test splits.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

147 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 131 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1608.00272 · arXiv 1608.00272

@misc{yu2016modeling,
  title = {{Modeling Context in Referring Expressions}},
  author = {Yu, Licheng and Poirson, Patrick and Yang, Shan and Berg, Alexander C. and Berg, Tamara L.},
  year = {2016},
  doi = {10.48550/arxiv.1608.00272},
  eprint = {1608.00272},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1608.00272}
}

References

41 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bell, S., Zitnick, C.L., Bala, K., Girshick, R.: Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks (2015) find
  2. Brown-Schmidt, S., Tanenhaus, M.K.: Watching the eyes when talking about size: An investigation of message formulation and utterance planning. Journal of Memory and Language 54(4), 592–609 (2006) find
  3. Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual recognition and description. In: CVPR (2015) find our card
  4. Erhan, D., Szegedy, C., Toshev, A., Anguelov, D.: Scalable object detection using deep neural networks. In: CVPR (2014) find
  5. Fang, H., Gupta, S., Iandola, F., Srivastava, R.K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J.C., et al.: From captions to visual concepts and back. In: CVPR (2015) find
  6. Farhadi, A., Hejrati, M., Sadeghi, M.A., Young, P., Rashtchian, C., Hockenmaier, J., Forsyth, D.: Every picture tells a story: Generating sentences from images. In: ECCV (2010) find
  7. FitzGerald, N., Artzi, Y., Zettlemoyer, L.S.: Learning distributions over logical forms for referring expression generation. In: EMNLP. pp. 1914–1925 (2013) find
  8. Funakoshi, K., Watanabe, S., Kuriyama, N., Tokunaga, T.: Generating referring expressions using perceptual groups. In: Natural Language Generation, pp. 51–60. Springer (2004) find
  9. Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1440–1448 (2015) find
  10. Greff, K., Srivastava, R.K., Koutnı́k, J., Steunebrink, B.R., Schmidhuber, J.: Lstm: A search space odyssey. arXiv preprint arXiv:1503.04069 (2015) find
  11. Grice, H.P.: Logic and conversation. In: Cole, P., Morgan, J.L. (eds.) Syntax and Semantics: Vol. 3: Speech Acts, pp. 41–58. Academic Press, San Diego, CA (1975) find
  12. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015) find
  13. Hodosh, M., Young, P., Hockenmaier, J.: Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research (2013) find
  14. Hu, R., Xu, H., Rohrbach, M., Feng, J., Saenko, K., Darrell, T.: Natural language object retrieval. In: CVPR (2016) find
  15. Johnson, J., Karpathy, A., Fei-Fei, L.: Densecap: Fully convolutional localization networks for dense captioning. arXiv preprint arXiv:1511.07571 (2015) — [arXiv:1511.07571](https://arxiv.org/abs/1511.07571) find
  16. Jordan, P., Walker, M.: Learning attribute selections for non-pronominal expressions. In: ACL (2000) find
  17. Jun Zhu, X.C., Yuille, A.: Deepm: A deep part-based model for object detection and semantic part localization (2015) find
  18. Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: CVPR (2015) find our card
  19. Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.L.: Referitgame: Referring to objects in photographs of natural scenes. In: EMNLP. pp. 787–798 (2014) find
  20. Kelleher, J.D., Kruijff, G.J.M.: Incremental generation of spatial referring expressions in situated dialog. In: ACL (2006) find
  21. Kiros, R., Salakhutdinov, R., Zemel, R.S.: Unifying visual-semantic embeddings with multimodal neural language models. TACL (2015) find our card
  22. Krahmer, E., Van Deemter, K.: Computational generation of referring expressions: A survey. Computational Linguistics 38(1), 173–218 (2012) find
  23. Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A.C., Berg, T.: Babytalk: Understanding and generating simple image descriptions. Pattern Analysis and Machine Intelligence, IEEE Transactions on (2013) find
  24. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: ECCV (2014) find our card
  25. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S.: Ssd: Single shot multibox detector (2015) find
  26. Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., Murphy, K.: Generation and comprehension of unambiguous object descriptions. In: CVPR (2016) find our card
  27. Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., Yuille, A.: Deep captioning with multimodal recurrent neural networks (m-rnn). ICLR (2015) find
  28. Mitchell, M., van Deemter, K., Reiter, E.: Natural reference to objects in a visual domain. In: Proceedings of the 6th international natural language generation conference. pp. 95–104. Association for Computational Linguistics (2010) find
  29. Mitchell, M., Reiter, E., van Deemter, K.: Typicality and object reference. Cognitive Science (CogSci) (2013) find
  30. Mitchell, M., Van Deemter, K., Reiter, E.: Generating expressions that refer to visible objects. In: HLT-NAACL. pp. 1174–1184 (2013) find
  31. Ordonez, V., Kulkarni, G., Berg, T.L.: Im2text: Describing images using 1 million captioned photographs. In: Advances in Neural Information Processing Systems (2011) find
  32. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: NIPS (2015) find our card
  33. Rohrbach, A., Rohrbach, M., Hu, R., Darrell, T., Schiele, B.: Grounding of textual phrases in images by reconstruction. arXiv preprint arXiv:1511.03745 (2015) — [arXiv:1511.03745](https://arxiv.org/abs/1511.03745) find
  34. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115(3), 211–252 (2015) find our card
  35. Sadeghi, F., Zitnick, C.L., Farhadi, A.: Visalogy: Answering visual analogy questions. In: NIPS (2015) find
  36. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014) find our card
  37. Socher, R., Karpathy, A., Le, Q.V., Manning, C.D., Ng, A.Y.: Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics (2014) find our card
  38. Viethen, J., Dale, R.: The use of spatial relations in referring expression generation. In: Proceedings of the Fifth International Natural Language Generation Conference. pp. 59–67. Association for Computational Linguistics (2008) find
  39. Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: CVPR (2015) find our card
  40. Winograd, T.: Understanding natural language. Cognitive psychology 3(1), 1–191 (1972) find
  41. Xu, K., Ba, J., Kiros, R., Courville, A., Salakhutdinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. ICML (2015) find our card