Elamur Research⌕
☰
home / papers / 2019 / Characterizing the dynamics of learning in repeated reference games

Characterizing the dynamics of learning in repeated reference games

Paper · 2019 · Robert D. Hawkins, Michael C. Frank and Noah D. Goodman

Summary

When two people repeatedly talk about the same hard-to-name objects, their wording changes over time. Hawkins, Frank and Goodman study this process with a large online version of the classic tangram reference game. One player, the director, describes a target shape. The other, the matcher, has to pick it out of twelve, and every shape comes back as the target several times. The authors release an open corpus of more than 15,000 chat utterances from both a free-matching and a cued variant of the game. They then use NLP tools to analyse it: part-of-speech tags and syntax trees to track which structures disappear, and GloVe embeddings to track meaning. The findings are:

  • Each pair settles on its own stable and efficient labels.
  • Words that were discriminative early on tend to survive.
  • Shortening is not random. Whole clauses and modifiers drop out together, often after the listener signals understanding, and short open-class labels remain.

This gives quantitative targets for computational models of how conventions form.

Key points

  • 480 Mechanical Turk workers took part, with 218 playing the free-matching variant and 262 playing the cued one. The cued data (83 pairs) is the pre-registered confirmatory sample, analysis plan at osf.io/vzvmf. The classic Clark & Wilkes-Gibbs (1986) study had 8 pairs.
  • Each trial showed a 6 × 2 grid of the twelve tangrams from Clark & Wilkes-Gibbs, with an unrestricted text chat.
  • The cued design links every utterance to a known target object, which allows analysis object by object. The free version required location talk and produced messages that mixed several targets.
  • Shortening was compared with a "random corruption" baseline, in which each word is dropped independently. Real reduction removes syntactic units in clusters and depends on listener feedback.
  • Arbitrariness and stability were measured as embedding similarity across pairs and across repetitions. Both signatures grow as the interaction goes on.

Interesting ideas

  • I1. Testing reduction against a simple word-dropping null model shows whether shortening follows structure; raw length counts cannot.
  • I2. A permutation baseline that scrambles words across targets before averaging embeddings rules out artefacts caused by utterance length.
  • I3. A word's discriminativeness in the first context (used for one target more than the others) predicts whether it becomes the lasting label.
  • I4. Speakers shorten after positive feedback even when they cannot be interrupted, which treats the reply as evidence of understanding.
  • I5. After repeated use, a long description collapses to a short open-class label such as an adjective plus a noun.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Measure whether dropped material corresponds to whole UD subtrees (acl, amod, nmod) rather than scattered tokens, and use that as a check on how parse quality affects analyses of reduction.
  • I2 (learn): Add similar scrambled-input negative controls whenever similarity scores between texts are compared, so length effects are not mistaken for meaning.
  • I3 (world): When tracking entities across a text, prefer the earliest distinctive descriptor as the entity's canonical name, so later short mentions can be linked back to it.
  • I4 (frames): Model acknowledgement speech acts in dialogue as updates to common ground that license shorter, partner-specific referring expressions in later turns.
  • I5 (back): In generation, refer to an entity again with its established short label instead of repeating the full description, making text from the meaning graph more natural.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States310 works · 284,480 citations🎓Princeton UniversityPrinceton · United States61 works · 36,471 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 worksCultural Studies · 10 works

Cite

Cite as Robert D. Hawkins, Michael C. Frank and Noah D. Goodman. 2019. Characterizing the dynamics of learning in repeated reference games. arXiv:1912.07199.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1912.07199 · arXiv 1912.07199

@misc{hawkins2019characterizing,
  title = {{Characterizing the dynamics of learning in repeated reference games}},
  author = {Robert D. Hawkins, Michael C. Frank and Noah D. Goodman},
  year = {2019},
  doi = {10.48550/arxiv.1912.07199},
  eprint = {1912.07199},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1912.07199}
}

References

54 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arora, S., Liang, Y., & Ma, T. (2017). A simple but tough-to-beat baseline for sentence embeddings. In International Conference on Learning Representations (ICLR). find
  2. Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). A neural probabilistic language model. Journal of machine learning research, 3 , 1137–1155.
  3. Brennan, S. E., & Clark, H. H. (1996). Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22 , 1482. ↗
  4. Brennan, S. E., & Hanna, J. E. (2009). Partner-specific adaptation in dialog. Topics in Cognitive Science, 1 . ↗
  5. Brochhagen, T. (2017). Signalling under Uncertainty: Interpretative Alignment without a Common Prior. The British Journal for the Philosophy of Science, . URL: https://doi.org/10.1093/bjps/axx058. doi:10.1093/bjps/axx058. ↗
  6. Carroll, J. M. (1980). Naming and describing in social communication. Language and Speech, 23 , 309–322. ↗
  7. Choi, J. D., & Palmer, M. (2012). Guidelines for the clear style constituent to dependency conversion. Technical Report. find
  8. Clark, H. H. (1983). Making sense of nonce sense. The process of language understanding, (pp. 297–331). find
  9. Clark, H. H. (1998). Communal lexicons. In K. Malmkjaer, & J. Williams (Eds.), Context in language learning and language understanding chapter 4. (pp. 63–87). Cambridge: Cambridge University Press. find
  10. Clark, H. H., & Wilkes-Gibbs, D. (1986). Referring as a collaborative process. Cognition, 22 , 1–39. ↗
  11. Corbett, G. G., Fraser, N. M., McGlashan, S. et al. (1993). Heads in grammatical theory. Cambridge University Press. find
  12. Davidson, D. (1986). A nice derangement of epitaphs. Philosophical grounds of rationality: Intentions, categories, ends, 4 , 157–174. find
  13. Delaney-Busch, N., Morgan, E., Lau, E., & Kuperberg, G. R. (2019). Neural evidence for bayesian trial-by-trial adaptation on the n400 during semantic priming. Cognition, 187 , 10–20. ↗
  14. Ficler, J., & Goldberg, Y. (2017). conjunction-specific features. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers (pp. 343–348). Valencia, Spain: Association for Computational Linguistics. find
  15. Frank, M. C. (2018). With great data comes great (theoretical) opportunity. Trends in Cognitive Sciences, 22 , 669–671. ↗
  16. Fussell, S. R., & Krauss, R. M. (1989). The effects of intended audience on message production and comprehension: Reference in a common ground framework. Journal of Experimental Social Psychology, 25 , 203–219. Improving a strong neural parser with Garrod, S., Fay, N., Lee, J., Oberlander, J., & MacLeod, T. (2007). Foundations of representation: where might graphical symbol systems come from? Cognitive Science, 31 , 961–987. ↗
  17. Gibson, E., Futrell, R., Jara-Ettinger, J., Mahowald, K., Bergen, L., Ratnasingam, S., Gibson, M., Piantadosi, S. T., & Conway, B. R. (2017). Color naming across languages reflects color use. Proceedings of the National Academy of Sciences, 114 , 10785–10790. ↗
  18. Gibson, E., Futrell, R., Piandadosi, S. T., Dautriche, I., Mahowald, K., Bergen, L., & Levy, R. (2019). How efficiency shapes human language. Trends in cognitive sciences, . ↗
  19. Glucksberg, S., & McGlone, M. S. (2001). Understanding figurative language: From metaphor to idioms. 36. Oxford University Press on Demand. find
  20. Goodman, N. D., & Frank, M. C. (2016). Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences, 20 , 818 – 829. ↗
  21. Hawkins, R. X. D. (2015). Conducting real-time multiplayer experiments on the web. Behavior Research Methods, 47 , 966–976. ↗
  22. Hawkins, R. X. D., Frank, M. C., & Goodman, N. D. (2017). Convention-formation in iterated reference games. In Proceedings of the 39th annual meeting of the cognitive science society. ↗
  23. Hawkins, R. X. D., Goodman, N. D., & Goldstone, R. L. (2019). The emergence of social norms and conventions. Trends in cognitive sciences, 23 , 158–169. ↗
  24. Honnibal, M., & Johnson, M. (2015). An improved non-monotonic transition system for dependency parsing. In Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 1373–1378).
  25. Honnibal, M., & Montani, I. (2019). spaCy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear , . find
  26. Hudson, R. A. (1984). Word grammar . Blackwell Oxford. find
  27. Hupet, M., & Chantraine, Y. (1992). Changes in repeated references: Collaboration or repetition effects? 485–496. ↗
  28. Jurafsky, D., & Martin, J. H. (2014). Speech and language processing volume 3. Pearson London. find
  29. Kiperwasser, E., & Goldberg, Y. (2016). Simple and accurate dependency parsing using bidirectional lstm feature representations. Transactions of the Association for Computational Linguistics, 4 , 313–327. ↗
  30. Kirby, S., Tamariz, M., Cornish, H., & Smith, K. (2015). Compression and communication in the cultural evolution of linguistic structure. Cognition, 141 , 87–102. ↗
  31. Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., & Fidler, S. (2015). Skip-thought vectors. In Advances in neural information processing systems (pp. 3294–3302). ↗
  32. Kleinschmidt, D. F. (2019). Structure in talker variability: How much is there and how much can it help? Language, cognition and neuroscience, 34 , 43–68. ↗
  33. Krauss, R. M., Garlock, C. M., Bricker, P. D., & McMahon, L. E. (1977). The role of audible and visible back-channel responses in interpersonal communication. Journal of personality and social psychology, 35 , 523. ↗
  34. Krauss, R. M., & Weinheimer, S. (1964). Changes in reference phrases as a function of frequency of usage in social interaction: A preliminary study. Psychonomic Science, 1 , 113–114. ↗
  35. Krauss, R. M., & Weinheimer, S. (1966). Concurrent feedback, confirmation, and the encoding of referents in verbal communication. Journal of Personality and Social Psychology, 4 , 343. Journal of psycholinguistic research, 21 , Kübler, S., McDonald, R., & Nivre, J. (2009). Dependency parsing. Synthesis lectures on human language technologies, 1 , 1–127. ↗
  36. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological review , 104 , 211. ↗
  37. Lascarides, A., & Copestake, A. (1998). Pragmatics and word meaning. Journal of linguistics, 34 , 387–414. find
  38. Levay, K. E., Freese, J., & Druckman, J. N. (2016). and political composition of mechanical turk samples. 2158244016636433. find
  39. Lewis, D. (1969). Convention: A philosophical study. Harvard University Press. find
  40. Maaten, L. v. d., & Hinton, G. (2008). Visualizing data using t-sne. Journal of machine learning research, 9 , 2579–2605. find
  41. Manning, C. D. (2011). Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In International conference on intelligent text processing and computational linguistics (pp. 171–189). Springer.
  42. Markman, A. B., & Makin, V. S. (1998). Referential communication and category acquisition. Journal of Experimental Psychology: General , 127 , 331. ↗
  43. Metzing, C., & Brennan, S. E. (2003). When conceptual pacts are broken: Partner-specific effects on the comprehension of referring expressions. Journal of Memory and Language, 49 , 201–213. ↗
  44. Nölle, J., Staib, M., Fusaroli, R., & Tylén, K. (2018). The emergence of systematicity: how environmental and communicative factors shape a novel communication system. Cognition, 181 , 93–104. ↗
  45. Osgood, C. E. (1952). The nature and measurement of meaning. Psychological bulletin, 49 , 197. The demographic Sage Open, 6 , Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532–1543). ↗
  46. Pickering, M. J., & Garrod, S. (2004). Toward a mechanistic psychology of dialogue. Behavioral and brain sciences, 27 , 169–190. ↗
  47. Prelec, D. (2004). A bayesian truth serum for subjective data. science, 306 , 462–466. find
  48. Robertson, S. (2004). Understanding inverse document frequency: on theoretical arguments for idf. Journal of documentation, 60 , 503–520. ↗
  49. Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information processing & management, 24 , 513–523. ↗
  50. Schober, M. F., & Clark, H. H. (1989). Understanding by addressees and overhearers. Cognitive Psychology, 21 , 211–232. ↗
  51. Smith, N. J., Goodman, N. D., & Frank, M. (2013). Learning and using language via recursive pragmatic reasoning about other agents. In Advances in neural information processing systems (pp. 3039–3047). find
  52. Sparck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28 , 11–21. ↗
  53. Wilkes-Gibbs, D., & Clark, H. H. (1992). Coordinating beliefs in conversation. Journal of memory and language, 31 , 183–194. ↗
  54. Winters, J., Kirby, S., & Smith, K. (2014). Languages adapt to their contextual niche. Language and Cognition, (pp. 1–35). ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hawkins-2019-characterizing-dynamics-learning-repeated-reference/