Elamur Research⌕
☰
home / papers / 2014 / ReferItGame: Referring to Objects in Photographs of Natural Scenes

ReferItGame: Referring to Objects in Photographs of Natural Scenes

Summary

The paper studies referring expression generation (REG), which means producing phrases that single out one object in an image. Earlier REG corpora used synthetic graphics or small objects placed on plain backgrounds. Here the setting is cluttered, real-world photographs. To get data cheaply and at scale, the authors built ReferItGame, a two-player online game. One player writes a description of an outlined object, and the other must click on it, so each successful round also checks the expression. Using images from the ImageCLEF dataset, the game produced 130,525 verified expressions for 96,654 objects in 19,894 photos. Their analysis shows that an object's category shapes how people describe it, including which attributes they mention, which nearby objects they use as anchors, and which category names they choose. Building on this, they propose an optimization model that decides jointly which attributes to mention and which values to give them. It combines visual estimates with category-specific priors. The model clearly beats a prior-only baseline on three test sets. The resource and analysis move REG toward realistic visual scenes.

Key points

  • Corpus size: 130,525 expressions referring to 96,654 distinct objects across 19,894 natural-scene photographs, drawn from ImageCLEF. Over 100k distinct games were created, one per labeled object.
  • Expressions from rounds where the second player clicked the wrong object are released but left out of the analyses.
  • Relative references, such as locating a car by where it sits next to a man, become more frequent as more objects of the same type appear in the image. The probability is 0.12 with one such object, 0.22 with two, and 0.26 with more.
  • The generation model is solved as a binary integer linear program with IBM ILOG CPLEX. It uses unary priors that mix category-specific and global counts, plus pairwise potentials for attribute co-occurrence and for how well relative-location and relative-object words fit together.
  • Evaluation uses three test sets of 500 objects each, with 3 manually attribute-labeled expressions per object. Human agreement is 72.31%. The full model reaches precision/recall of 36.28/53.44 (A), 36.68/59.80 (B) and 37.73/48.54 (C), compared with baselines of about 28–30% precision and 37–51% recall.

Interesting ideas

  • I1. Built-in verification through a second player: an annotation is kept only if another person can act on it correctly. This removes a separate validation step and could be reused for collecting instructions, place descriptions, or paraphrases.
  • I2. Canned partner when nobody is online: a solo player is matched against pre-recorded sessions. Their text is compared to stored expressions using lemmatized bag-of-words cosine similarity, and the stored click is replayed. This cheap trick keeps a crowdsourcing game alive during low-traffic periods.
  • I3. Prioritizing the hardest items first: games were ordered so that objects sharing a category with others in the same image came first. Collection effort is focused where disambiguation is actually needed.
  • I4. Category-conditioned priors with global backoff: attribute usage is estimated per object category and smoothed toward dataset-wide counts. This is a simple way to capture domain-specific language habits without losing coverage for rare categories.
  • I5. Stratified test sets: the model was evaluated on a random sample, on a category-balanced sample of interesting categories, and on images with several same-category objects. Comparing these shows how performance depends on the data distribution, not just a single score.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

228 papers of our library cite it (of the 140884 whose bibliographies we read); 32 of them have a page here, 196 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/d14-1086

@inproceedings{kazemzadeh2014referitgame,
  title = {{ReferItGame: Referring to Objects in Photographs of Natural Scenes}},
  author = {Kazemzadeh, Sahar and Ordonez, Vicente and Matten, Mark and Berg, Tamara},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1086},
  url = {https://doi.org/10.3115/v1/d14-1086}
}

References

42 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmet Aker and Robert Gaizauskas. 2010. Generating image descriptions using dependency relational patterns. In Association for Computational Linguistics (ACL). find
  2. Andrei Barbu, Alexander Bridge, Zachary Burchill, Dan Coroian, Sven J. Dickinson, Sanja Fidler, Aaron Michaux, Sam Mussman, Siddharth Narayanaswamy, Dhaval Salvi, Lara Schmidt, Jiangnan Shangguan, Jeffrey Mark Siskind, Jarrell W. Waggoner, Song Wang, Jinlian Wei, Yifan Yin, and Zhiqi Zhang. 2012. Vi… find
  3. Robert Dale and Ehud Reiter. 1995. Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive Science (CogSci), 19:233264. find our card
  4. Robert Dale and Ehud Reiter. 2000. Building natural language generation systems. In Cambridge University Press. find
  5. Jia Deng, Alexander C. Berg, Kai Li, and Fei-Fei Li. 2010. What does classifying more than 10,000 image categories tell us? In European Conference on Computer Vision (ECCV). find
  6. Jia Deng, Alex Berg, Sanjeev Satheesh, Hao Su, Aditya Khosla, and Fei-Fei Li. 2012. Large scale visual recognition challenge. In http://www.image-net.org/challenges/LSVRC/2012/index. ↗
  7. Jia Deng, Jonathan Krause, and Li Fei-Fei. 2013. Fine-grained crowdsourcing for fine-grained recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  8. Hugo Jair Escalante, Carlos A. Hernandez, Jesus A. Gonzalez, A. Lopez-Lopez, Manuel Montes, Eduardo F. Morales, L. Enrique Sucar, Luis Villasenor, and Michael Grubinger. 2010. The segmented and annotated iapr tc-12 benchmark. Computer Vision and Image Understanding (CVIU). find
  9. Rui Fang, Changsong Liu, Lanbo She, and Joyce Chai. 2013. Towards situated dialogue: Revisiting referring expression generation. In Empirical Methods on Natural Language Processing (EMNLP). find
  10. Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010. Every picture tells a story: generating sentences for images. In European Conference on Computer Vision (ECCV). find
  11. Yansong Feng and Mirella Lapata. 2010. How many words is a picture worth? automatic caption generation for news images. In Association for Computational Linguistics (ACL). find
  12. Yansong Feng and Mirella Lapata. 2013. Automatic caption generation for news images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(4):797–812. find
  13. Nicholas FitzGerald, Yoav Artzi, and Luke Zettlemoyer. 2013. Learning distributions over logical forms for referring expression generation. In Empirical Methods on Natural Language Processing (EMNLP). find
  14. H. Paul Grice. 1975. Logic and conversation. page 4158. find
  15. Michael Grubinger, Paul D. Clough, Henning Muller, and Thomas Deselaers. 2006. The iapr benchmark: A new evaluation resource for visual information systems. In Proceedings of the International Workshop OntoImage (LREC). find
  16. Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Raymond Mooney, Trevor Darrell, and Kate Saenko. 2013. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In International Conference on Computer Vi… find
  17. Emiel Krahmer and Kees van Deemter. 2012. Computational generation of referring expressions: A survey. In Computational Linguistics, volume 38, page 173218. find our card
  18. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Neural Information Processing Systems (NIPS). find
  19. Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. 2011. Babytalk: Understanding and generating simple image descriptions. In IEEE Computer Vision and Pattern Recognition (CVPR). find
  20. Polina Kuznetsova, Vicente Ordonez, Alex Berg, Tamara L Berg, and Yejin Choi. 2012. Collective generation of natural image descriptions. In Association for Computational Linguistics (ACL). find
  21. Margaret Mitchell, Kees van Deemter, and Ehud Reiter. 2010. Natural reference to objects in a visual domain. In International Natural Language Generation Conference (INLG). find
  22. Margaret Mitchell, Kees van Deemter, and Ehud Reiter. 2011. Two approaches for generating size modifiers. In European Workshop on Natural Language Generation. find
  23. Margaret Mitchell, Ehud Reiter, and Kees van Deemter. 2013a. Typicality and object reference. In Cognitive Science (CogSci). find
  24. Margaret Mitchell, Kees van Deemter, and Ehud Reiter. 2013b. Generating expressions that refer to visible objects. In North American Chapter of the Association for Computational Linguistics (NAACL). find
  25. Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. 2011. Im2text: Describing images using 1 million captioned photographs. In Neural Information Processing Systems (NIPS). find
  26. Vicente Ordonez, Jia Deng, Yejin Choi, Alexander C. Berg, and Tamara L. Berg. 2013. From large scale image categorization to entry-level categories. In International Conference on Computer Vision (ICCV). find
  27. Florent Perronnin, Zeynep Akata, Zaid Harchaoui, and Cordelia Schmid. 2012. Towards good practice in large-scale learning for image classification. In Computer Vision and Pattern Recognition (CVPR). find
  28. Yuan Ren, Kees Van Deemter, and Jeff Z Pan. 2010. Charting the potential of description logic for the generation of referring expressions. In International Natural Language Generation Conference (INLG). find
  29. Eleanor Rosch. 1978. Principles of categorization. Cognition and Categorization, page 2748. find
  30. Nitin Seemakurty, Jonathan Chu, Luis von Ahn, and Anthony Tomasic. 2010. Word sense disambiguation via human computation. In Human Computation Workshop. find
  31. Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013. Parsing With Compositional Vector Grammars. In Association for Computational Linguistics (ACL). find
  32. Kees van Deemter, Ielka van der Sluis, and Albert Gatt. 2006. Building a semantically transparent corpus for the generation of referring expressions. In International Conference on Natural Language Generation (INLG). find
  33. Kees Van Deemter, Albert Gatt, Roger PG van Gompel, and Emiel Krahmer. 2012. Toward a computational psycholinguistics of reference production. In Topics in Cognitive Science, volume 4(2), page 166183. find
  34. Jette Viethen and Robert Dale. 2008. The use of spatial relations in referring expression generation. In International Natural Language Generation Conference (INLG). find
  35. Jette Viethen and Robert Dale. 2010. Speaker-dependent variation in content selection for referring expression generation. In Australasian Language Technology Workshop. find
  36. Jette Viethen, Margaret Mitchell, and Emiel Krahmer. 2013. Graphs and spatial relations in the generation of referring expressions. In European Workshop on Natural Language Generation. find
  37. Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In ACM Conf. on Human Factors in Computing Systems (CHI). find
  38. Luis von Ahn, Mihir Kedia, and Manuel Blum. 2006a. Verbosity: A game for collecting common-sense knowledge. In ACM Conference on Human Factors in Computing Systems (CHI). find
  39. Luis von Ahn, Ruoran Liu, and Manuel Blum. 2006b. Peekaboom: A game for locating objects in images. In ACM Conference on Human Factors in Computing Systems (CHI). find
  40. Terry Winograd. 1972. Understanding natural language. Cognitive Psychology, 3(1):1191. find
  41. Yezhou Yang, Ching Lik Teo, Hal Daume III, and Yiannis Aloimonos. 2011. Corpus-guided sentence generation of natural images. In Empirical Methods on Natural Language Processing (EMNLP). find
  42. Benjamin Z. Yao, Xiong Yang, Liang Lin, Mun Wai Lee, and Song-Chun Zhu. 2010. I2t: Image parsing to text description. Proc. IEEE, 98(8). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kazemzadeh-2014-referitgame-referring-objects-photographs-natural/