Elamur Research⌕
☰
home / papers / 2019 / A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Context

A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Context

Paper · 2019 · Takuma Udagawa and Akiko Aizawa

Summary

Dialogue systems rarely have to build mutual understanding (common ground) from scratch. Existing benchmarks either give both partners the same complete view or describe the world with discrete categories that leave little room for ambiguity. The authors design a small cooperative game to make grounding genuinely hard but still easy to score. Two players each see 7 dots on a 2-D plane. Their views overlap only partly, and every attribute (position, size, shade of gray) varies continuously. They chat until each picks a dot, and they win only if both pick the same one. Paired crowdworkers produced 6,760 dialogues through a controlled collection protocol. The resulting corpus contains hedged descriptions, clarification questions and partial acknowledgements, and it also shows a non-linguistic tendency to pick dots that stand out to both players. On a target-selection subtask, simple neural baselines that read the dialogue clearly beat context-only ones, yet they stay far below humans. The game therefore works as a cheap, measurable testbed for grounding skills.

Key points

  • The corpus has 6,760 dialogues with over 32K utterances, collected on Amazon Mechanical Turk. Each pair got 20 seconds to study an example and up to 6 minutes to finish. Data and scripts are released as the onecommon repository (Alab-NII on GitHub).
  • Each player sees exactly 7 entities inside a fixed radius around their own position. Because of this design, success reduces to a 7-way classification problem scored by accuracy.
  • Collection rules aimed at interpretability, variety and reliability. Players had to alternate turns to avoid cross-talk and were told to stay on task with no chit-chat. The collection procedure follows the MutualFriends dataset, which serves as the comparison point.
  • Baselines encode the context with either an MLP or a Relation Network over all 21 entity pairs, and encode the dialogue with a bidirectional GRU. The data split is 8:1:1, and each model is trained with 10 seeds.
  • Accuracy on the full test set rises from 27.90 (context only, MLP) to 43.09 (context + dialogue, RN). On the success-only subset the best model reaches 49.44, while humans reach 82.50 on the uncorrelated set and 90.79 on the success-only set.

Interesting ideas

  • I1. The authors train a model that sees only the scene and never the dialogue, which measures how much of the score comes from selection bias rather than language understanding.
  • I2. They derive extra test sets by removing items that share the same dialogue with a different view, and by keeping only cases where the players succeeded. This separates model errors from noise in the human data.
  • I3. Continuous attributes push speakers toward relative, graded wording such as "darker" or "slightly left", which a symbolic system must resolve against actual values.
  • I4. The dialogues contain explicit grounding moves: a tentative description, a hypothesis-testing question, and a qualified "yes, slightly". These moves form small units that can be analysed.
  • I5. The Relation Network adds a feature for every pair of entities, so relations like "under" or "left of" become directly available to the model.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Every reader evaluation could include a matching "no text" or "shuffled text" control, so that gains are credited only when parsing actually adds information.
  • I2 (learn): Benchmarks could carry filtered slices (deduplicated, gold-consistent only) to show whether failures come from Mova itself or from flawed reference answers.
  • I3 (world): Comparative and hedged adjectives could be compiled into ordering constraints over measures on entities, rather than fixed categories, so that "the larger one" picks out a referent.
  • I4 (frames): Speech-act tagging could distinguish proposals, clarification requests and partial confirmations, and could update the confidence of a shared referent instead of treating each assertion as settled.
  • I5 (world): The text world could store explicit pairwise spatial or comparative relations between entities, making relational questions answerable and the chain of reasoning traceable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan45 works · 8,692 citations🔬National Institute of InformaticsTokyo · Japan21 works · 3,045 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Takuma Udagawa and Akiko Aizawa. 2019. A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Context. arXiv:1907.03399.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1907.03399 · arXiv 1907.03399

@misc{udagawa2019natural,
  title = {{A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Context}},
  author = {Takuma Udagawa and Akiko Aizawa},
  year = {2019},
  doi = {10.48550/arxiv.1907.03399},
  eprint = {1907.03399},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1907.03399}
}

References

30 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Austin, J. L. 1962. How to do things with words. Oxford university press. find
  2. Bordes, A., and Weston, J. 2016. Learning end-to-end goal-oriented dialog. CoRR abs/1605.07683. ↗
  3. Bunt, H.; Petukhova, V.; Traum, D.; and Alexandersson, J. 2017. Dialogue act annotation with the ISO 24617-2 standard. In Multimodal Interaction with W3C Standards. Springer. 109– 135. ↗
  4. Cho, K.; van Merrienboer, B.; Bahdanau, D.; and Bengio, Y. 2014. On the properties of neural machine translation: Encoder–decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, 103–111. Association for Computational Linguistics. our card
  5. Clark, H. H. 1996. Using language. Cambridge university press. find
  6. Das, A.; Kottur, S.; Gupta, K.; Singh, A.; Yadav, D.; Moura, J. M.; Parikh, D.; and Batra, D. 2017a. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 2. find
  7. Das, A.; Kottur, S.; Moura, J. M.; Lee, S.; and Batra, D. 2017b. Learning cooperative visual dialog agents with deep reinforcement learning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). find
  8. De Vries, H.; Strub, F.; Chandar, S.; Pietquin, O.; Larochelle, H.; and Courville, A. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In Proc. of CVPR. ↗
  9. Fang, R.; Doering, M.; and Chai, J. Y. 2015. Embodied collaborative referring expression generation in situated human-robot interaction. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, HRI ’15, 271–278. New York, NY, USA: ACM. find
  10. Grice, H. P. 1957. Meaning. The philosophical review 66(3):377–388. find
  11. He, H.; Balakrishnan, A.; Eric, M.; and Liang, P. 2017. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1766–1776. Association for Computational Linguistics. find
  12. Kazemzadeh, S.; Ordonez, V.; Matten, M.; and Berg, T. 2014. ReferItGame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 787–798. Association for Computational Linguistics. our card
  13. Kingma, D. P., and Ba, J. 2014. Adam: A method for stochastic optimization. CoRR abs/1412.6980. ↗
  14. Lewis, M.; Yarats, D.; Dauphin, Y.; Parikh, D.; and Batra, D. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2443–2453. Association for Computational Linguistics. our card
  15. Lewis, D. 1969. Convention cambridge. Mass.: Harvard UP. find
  16. Liu, C.-W.; Lowe, R.; Serban, I.; Noseworthy, M.; Charlin, L.; and Pineau, J. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2122–2132. Association for Computational Linguistics. our card
  17. Liu, C.; Fang, R.; and Chai, J. 2012. Towards mediating shared perceptual basis in situated dialogue. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 140–149. Association for Computational Linguistics. ↗
  18. Loper, E., and Bird, S. 2002. NLTK: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics - Volume 1, ETMTNLP ’02, 63–70. Stroudsburg, PA, USA: Association for Computational Linguistics. ↗
  19. Monroe, W.; Hawkins, R. X. D.; Goodman, N. D.; and Potts, C. 2017. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics 5:325–338. ↗ our card
  20. Novikova, J.; Dušek, O.; Cercas Curry, A.; and Rieser, V. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2241–2252. Association for Computational Linguistics. our card
  21. Santoro, A.; Raposo, D.; Barrett, D. G.; Malinowski, M.; Pascanu, R.; Battaglia, P.; and Lillicrap, T. 2017. A simple neural network module for relational reasoning. In Advances in neural information processing systems, 4967–4976. ↗ our card
  22. Schelling, T. C. 1980. The strategy of conflict. Harvard university press. find
  23. Searle, J. R. 1969. Speech acts: An essay in the philosophy of language, volume 626. Cambridge university press. find
  24. Serban, I. V.; Sordoni, A.; Bengio, Y.; Courville, A.; and Pineau, J. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, 3776–3783. AAAI Press. ↗ our card
  25. Suhr, A.; Lewis, M.; Yeh, J.; and Artzi, Y. 2017. A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, 217–223. find
  26. Tomasello, M. 2009. Constructing a language. Harvard university press. find
  27. Traum, D. R. 1994. A computational theory of grounding in natural language conversation. Technical report, ROCHESTER UNIV NY DEPT OF COMPUTER SCIENCE. find
  28. Vinyals, O., and Le, Q. V. 2015. A neural conversational model. CoRR abs/1506.05869. ↗ our card
  29. Williams, J.; Raux, A.; and Henderson, M. 2016. The dialog state tracking challenge series: A review. Dialogue & Discourse 7(3):4–33. ↗
  30. Young, S.; Gašić, M.; Thomson, B.; and Williams, J. D. 2013. POMDP-based statistical spoken dialog systems: A review. Proceedings of the IEEE 101(5):1160–1179. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/udagawa-2019-natural-language-corpus-common-grounding/