Elamur Research⌕
☰
home / papers / 2019 / The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue

The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue

Summary

Current visually grounded dialogue systems tend to lose consistency across a conversation, partly because they do not exploit the shared history that human partners build up. To support work on this, the authors created PhotoBook, a crowdsourced English corpus of about 2,500 goal-oriented chats. In each game, two players see grids of six similar MS COCO photos and must work out, through free-form text chat, which of their three highlighted images their partner also sees. Neither player has a fixed role. Over five rounds, target images reappear, so players re-describe the same pictures and can reuse descriptions they agreed on earlier. The analysis shows that players become faster and more accurate as a game goes on. Their utterances also get shorter, and later descriptions differ strongly from both standard captions and the players' own first mentions. A baseline reference resolution model that draws on earlier mentions in a reference chain resolves later descriptions better than one that ignores them. The authors conclude that richer models of common ground are needed.

Key points

  • Collection and scale: Data was gathered with ParlAI on Amazon Mechanical Turk from 2,506 completed games played by 1,514 workers. The corpus contains 164,615 utterances, 130,322 actions, and a vocabulary of 11,805 tokens.
  • Image sets: 30 sets of similar MS COCO images were built by matching the categories of their two largest objects. These yield 60 unique game sets covering 360 images.
  • Task performance: A full game took 14.2 minutes on average. Players labelled about 28.62 of 30 targets correctly, and efficiency rose from round to round.
  • Models: The HISTORY model, which uses reference-chain context, beats NO-HISTORY, especially on abstract or non-standard expressions such as "strange." Removing visual input lowers precision by roughly 21 points and recall by roughly 14 points.
  • Availability: The data, extracted reference chains, and code are released at dmg-photobook.github.io.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

47 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 43 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1184 · arXiv 1906.01530

@inproceedings{haber2019photobook,
  title = {{The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue}},
  author = {Haber, Janosch and Baumgärtner, Tim and Takmaz, Ece and Gelderloos, Lieke and Bruni, Elia and Fernández, Raquel},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1184},
  eprint = {1906.01530},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1184}
}

References

30 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Luis von Ahn, Mihir Kedia, and Manuel Blum. 2006. Verbosity: A game for collecting common-sense facts. In Proceedings of the Conference on Human Factors in Computing Systems. find
  2. Anne H. Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, et al. 1991. The HCRC Map Task corpus. Language and Speech, 34(4):351–366. ↗
  3. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In Proceedings of ICCV. ↗
  4. Susan E. Brennan and Herbert H. Clark. 1996. Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22:1482–1493. ↗
  5. Sarah Brown-Schmidt, Si On Yoon, and Rachel Anna Ryskin. 2015. People as contexts in conversation. In Psychology of Learning and Motivation, volume 62, chapter 3, pages 59–99. Elsevier. ↗
  6. Herbert H. Clark. 1996. Using Language. ’Using’ Linguistic Books. Cambridge University Press. find
  7. Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22(1):1 – 39. ↗
  8. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual Dialog. In Proceedings of CVPR. find
  9. Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017a. Guesswhat?! visual object discovery through multi-modal dialogue. In Proc. of CVPR. ↗
  10. Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron Courville. 2017b. Modulating early visual processing by language. In Proceedings of NIPS. ↗
  11. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In Proc. of CVPR. find
  12. Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P Bigham. 2018. A Data-Driven Analysis of Workers’ Earnings on Amazon Mechanical Turk. In Proceedings of the Conference on Human Factors in Computing Systems. find
  13. He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. In Proceedings of ACL. find
  14. Nikolai Ilinykh, Sina Zarrieß, and David Schlangen. 2018. The task matters: Comparing image captioning and task-based dialogical image description. In Proceedings of INLG. find
  15. Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. 2014. ReferIt Game: Referring to Objects in Photographs of Natural Scenes. In Proceedings of EMNLP. find
  16. Diederik P. Kingma and Jimmy Ba. 2014. A method for stochastic optimization. abs/1412.6980. Adam: CoRR, Robert M. Krauss and Sidney Weinheimer. 1964. Changes in reference phrases as a function of frequency of usage in social interaction: a preliminary study. Psychonomic Science, 1(1):113–114. ↗
  17. Robert M. Krauss and Sidney Weinheimer. 1966. Concurrent feedback, confirmation, and the encoding of referents in verbal communication. Journal of Personality and Social Psychology, 4(3):343–346. ↗
  18. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73. ↗ our card
  19. Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of Psychology, 22 140:55–55. find
  20. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Proc. of ECCV. ↗ our card
  21. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In Proceedings of CVPR. find our card
  22. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS. ↗
  23. Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A Dialog Research Software Platform. In Proceedings of EMNLP. find our card
  24. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR. ↗
  25. Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of ICCV. ↗
  26. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. ↗
  27. Robert Stalnaker. 1978. Assertion. In P. Cole, editor, Pragmatics, volume 9 of Syntax and Semantics, pages 315–332. New York Academic Press. find
  28. Tokunaga Takenobu, Iida Ryu, Terai Asuka, and Kuriyama Naoko. 2012. The REX corpora: A collection of multimodal corpora of referring expressions in collaborative problem solving dialogues. In Proceedings of LREC. find
  29. Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. In Proceedings of CVPR. find
  30. Etsuko Yoshida. 2011. Referring expressions in English and Japanese: patterns of use in dialogue processing, volume 208. John Benjamins Publishing. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/haber-2019-photobook-dataset-building-common-ground/