Elamur Research⌕
☰
home / papers / 2021 / SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations

SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations

Paper · 2021 · Satwik Kottur, Seungwhan Moon, Alborz Geramifard et al.

Summary

Task-oriented virtual assistants increasingly need to reason about what the user sees as well as what they say. The earlier SIMMC benchmarks relied on simplistic visual contexts. SIMMC 2.0 addresses this with a larger, more realistic resource for shopping conversations in the fashion and furniture domains, set in cluttered, photo-realistic VR store scenes.

The authors build scenes in Unity by hand-crafting seed stores and then programmatically swapping semantically compatible assets. They capture random camera viewpoints and automatically extract scene graphs and bounding boxes. An agenda-based multimodal simulator, consisting of a goal generator, a user simulator and an assistant simulator, produces annotated dialog flows. Crowd workers then rewrite these flows into natural language.

The resulting corpus supports four benchmarks:

  • Multimodal Disambiguation
  • MM-Coref
  • MM-DST
  • Response Generation

A multi-task GPT-2 baseline reaches only about 37% coreference F1, versus about 86% on SIMMC 1.0. This gap shows that visual grounding in realistic, cluttered scenes remains a hard open problem.

Key points

  • Dataset size: 11,244 dialogs, 117,236 utterances, 1,566 scene snapshots. Each dialog has about 10.4 utterances and 19.7 objects in the scene on average. Snapshots showing fewer than 5 objects were filtered out.
  • Two-phase collection: simulated multimodal self-play first, then manual paraphrasing. This lowers annotation time and cost while keeping full ground-truth labels.
  • Data split: train 65%, dev 5%, dev-test 15%, and a hidden test-std set of 15%.
  • Baselines:
    • A GPT-2 model (12-layer, 117M parameters) that reads flattened scene metadata scores 73.9 disambiguation accuracy, 36.64 coref F1, DST F1 of 81.72 / 94.53 and 0.192 BLEU.
    • The image-based MTN model scores 0.217 BLEU.
  • Availability: code and data are released at github.com/facebookresearch/simmc2.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States158 works · 211,459 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 269 worksDialogue systems · 401 works

Cite

Cite as Satwik Kottur, Seungwhan Moon, Alborz Geramifard and Babak Damavandi. 2021. SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.401

@inproceedings{kottur2021simmc,
  title = {{SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations}},
  author = {Satwik Kottur, Seungwhan Moon, Alborz Geramifard and Babak Damavandi},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.401},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.401}
}

References

35 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In ICCV. find
  2. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the Conference on Empirical Methods in Natural Language Proc… find
  3. Guan-Lin Chao and Ian Lane. 2019. Bert-dst: Scalable end-to-end dialogue state tracking with bidirectional encoder representations from transformer. In Annual Conference of the International Speech Communication Association (INTERSPEECH). find
  4. Paul A. Crook, Satwik Kottur, Seungwhan Moon, Ahmad Beirami, Eunjoon Cho, Rajen Subba, and Alborz Geramifard. 2021. Situated interactive multimodal conversations (simmc) track at dstc9. AAAI DSTC9 Workshop. find
  5. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In CVPR. find
  6. Mihail Eric, Rahul Goel, Shachi Paul, Adarsh Kumar, Abhishek Sethi, Peter Ku, Anuj Kumar Goyal, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tur. 2019. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint arXiv:1907.01669. find
  7. Matthew Fisher, Daniel Ritchie, Manolis Savva, Thomas Funkhouser, and Pat Hanrahan. 2012. Example-based synthesis of 3d object arrangements. In ACM SIGGRAPH Asia 2012 papers, SIGGRAPH Asia ’12. find
  8. Matthew Fisher, Manolis Savva, Yangyan Li, Pat Hanrahan, and Matthias Nießner. 2015. Activity-centric scene synthesis for functional 3d scene modeling. ACM Transactions on Graphics (TOG), 34(6). find
  9. Shuyang Gao, Sanchit Agarwal Abhishek Seth and, Tagyoung Chun, and Dilek Hakkani-Ture. 2019. Dialog state tracking: A neural reading comprehension approach. In Special Interest Group on Discourse and Dialogue (SIGDIAL).
  10. Chiori Hori, Anoop Cherian, Tim K. Marks, and Florian Metze. 2018. Audio visual scene-aware dialog track in dstc8. DSTC Track Proposal. find
  11. Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. arXiv preprint arXiv:2005.00796. find
  12. Xin Huang, Chor Seng Tan, Yan Bin Ng, Wei Shi, Kheng Hui Yeo, Ridong Jiang, and Jung Jae Kim. 2021. Joint generation and bi-encoder for situated interactive multimodal conversations. find
  13. AAAI 2021 DSTC9 Workshop. find
  14. Younghoon Jeong, Se Jin Lee, Youngjoong Ko, and Jungyun Seo. 2021. Tom : End-to-end task-oriented multimodal dialog system with gpt-2. find
  15. AAAI 2021 DSTC9 Workshop. find
  16. Byoungjae Kim, Inkwon Lee, Yeonseok Jeong, Ko Youngjoong, Myoung-Wan Koo, and Jungyun Seo. 2021. Improving multimodal api prediction via adding dialog state and various multimodal gates. find
  17. AAAI 2021 DSTC9 Workshop. find
  18. Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2019. Clevr-dialog: A diagnostic dataset for multi-round reasoning in visual dialog. arXiv preprint arXiv:1903.03166. find
  19. Po-Nien Kung, Tse-Hsuan Yang, Chung-Cheng Chang, Hsin-Kai Hsu, Yu-Jia Liou, and Yun-Nung Chen. 2021. Multi-task learning for situated multi-domain end-to-end dialogue systems. find
  20. AAAI 2021 DSTC9 Workshop. find
  21. Hung Le, Doyen Sahoo, Nancy Chen, and Steven Hoi. 2019. Multimodal transformer networks for end-to-end video-grounded dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5612–5623. find
  22. Seungwhan Moon, Satwik Kottur, Paul A Crook, Ankita De, Shivani Poddar, Theodore Levin, David Whitney, Daniel Difranco, Ahmad Beirami, Eunjoon Cho, Rajen Subba, and Alborz Geramifard. 2020. Situated and interactive multimodal conversations. arXiv preprint arXiv:2006.01460. find
  23. Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D’Haro, Abhinav Rastogi, Yun-Nung Chen, Mihail Eric, Behnam Hedayatnia, Karthik Gopalakrishnan, Yang Liu, Chao-Wei Huang, et al. 2020. Overview of the ninth dialog system technology challenge: Dstc9. arXiv preprint arXiv:2011.06486. — [arXiv:2011.06486](https://arxiv.org/abs/2011.06486) find
  24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  25. Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIGDIAL), pages 263–272. find
  26. Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. 2020. Soloist: Few-shot task-oriented dialog with a single pre-trained auto-regressive model. arXiv preprint arXiv:2005.05298. find
  27. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  28. Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Association for the Advancement of Artificial Intelligence (AAAI). find
  29. Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a pomdp dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volum…
  30. Matteo Antonio Senese, Giuseppe Rizzo, Alberto Benincasa, and Barbara Caputo. 2021. A response retrieval approach for dialogue using a multi-attentive transformer. find
  31. AAAI 2021 DSTC9 Workshop. find
  32. Pararth Shah, Dilek Hakkani-Tür, Gokhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. 2018. Building a conversational agent overnight with dialogue self-play. arXiv preprint arXiv:1801.04871. find
  33. Unity Technologies. 2019. Unity. https://unity.com/. ↗
  34. Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. 2018. Talk the walk: Navigating new york city through grounded dialogue. arXiv preprint arXiv:1807.03367. — [arXiv:1807.03367](https://arxiv.org/abs/1807.03367) find
  35. Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In CVPR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kottur-2021-simmc-2-0-task-oriented/