Elamur Research
☰
home / papers / 2017 / Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding

Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/Q17-1023.pdf

Summary

The paper asks how a system can interpret color descriptions as people do, taking into account which other colors appear alongside the target. The authors ran dyadic reference games on Amazon Mechanical Turk. In each round, a speaker used a chat box to describe one of three color patches, and a listener clicked the patch they believed was meant. Rounds were balanced across close, split and far conditions, defined by perceptual color distance.

The authors trained a literal listener and a literal speaker as RNNs on the human utterances. They then linked the two through the Rational Speech Acts (RSA) framework, so that pragmatic agents reason recursively about each other. Unlike earlier RSA work, this avoids hand-written grammars and fixed message sets.

Combining the speaker and listener perspectives interprets descriptions more accurately than the base RNN listener alone. Pragmatic reasoning helps mainly in difficult contexts, such as very similar colors or targets that are hard to name. The approach shows that RSA-style pragmatics can scale to diverse, real language, and the released corpus supports research on grounded communication.

Key points

  • After filtering, the corpus contains 53,365 utterances from 46,994 rounds in 948 games. The raw collection came from 967 participants playing 1,059 games of 50 rounds each, and both the filtered and raw data are released.
  • Conditions were defined with the CIEDE color-difference standard (threshold θ = 20) and are roughly equally represented: 15,519 close, 15,693 split and 15,782 far trials.
  • Human listeners were about 97% accurate on far trials, 90% on split trials and 83% on close trials.
  • The models include a literal RNN listener L0 and speaker S0, a speaker-derived listener L1, and a two-level listener L2, plus blended listeners La and Le. The biggest gain comes from blending in the speaker perspective, even though the speaker-only listener performs poorly on its own.
  • Speakers, both human and model, use more words, comparatives, superlatives and negations when distractors are close to the target. Much of this pragmatic behavior seen in people is also captured by the speaker model built into the listener.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

61 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 55 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00064 · arXiv 1703.10186

@article{monroe2017colors,
  title = {{Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding}},
  author = {Monroe, Will and Hawkins, Robert X.D. and Goodman, Noah D. and Potts, Christopher},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2017},
  doi = {10.1162/tacl\_a\_00064},
  eprint = {1703.10186},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00064}
}

References

41 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods on Natural Language Processing (EMNLP), pages 1173–1182. find our card
  2. Bert Baumgaertner, Raquel Fernandez, and Matthew Stone. 2012. Towards a flexible semantics: Colour terms in collaborative reference tasks. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM), pages 80–84. find
  3. Leon Bergen, Roger Levy, and Noah D. Goodman. 2016. Pragmatic reasoning through semantic inference. Semantics and Pragmatics, 9(20). find
  4. Gillian Brown and George Yule. 1983. Discourse Analysis. Cambridge University Press. find
  5. Sarah Brown-Schmidt and Michael K. Tanenhaus. 2008. Real-time investigation of referential domains in unscripted conversation: A targeted language game approach. Cognitive Science, 32(4):643–684. find
  6. Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22(1):1– 39. 11 https://cocolab.stanford.edu/datasets/ colors.html Herbert H. Clark. 1996. Using Language. Cambridge University Press. ↗
  7. Richard S. Cook, Paul Kay, and Terry Regier. 2005. The World Color Survey database. Handbook of Categorization in Cognitive Science, pages 223–241. find
  8. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. MIT Press. find
  9. Michael C. Frank and Noah D. Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998. find our card
  10. Dave Golland, Percy Liang, and Dan Klein. 2010. A game-theoretic approach to generating spatial descriptions. In Proceedings of the 2010 Conference on Empirical Methods on Natural Language Processing (EMNLP), pages 410–419. find
  11. Noah D. Goodman and Michael C. Frank. 2016. Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences, 20(11):818–829. ↗
  12. Noah D. Goodman and Andreas Stuhlmüller. 2013. Knowledge and implicature: Modeling language understanding as social cognition. Topics in Cognitive Science, 5(1):173–184. find
  13. H. Paul Grice. 1975. Logic and conversation. In Peter Cole and Jerry Morgan, editors, Syntax and Semantics, Volume 3: Speech Acts, pages 43–58. Academic Press. find
  14. Joy E. Hanna, Michael K. Tanenhaus, and John C. Trueswell. 2003. The effects of common ground and perspective on domains of referential interpretation. Journal of Memory and Language, 49(1):43–61. find
  15. Robert X. D. Hawkins. 2015. Conducting real-time multiplayer experiments on the web. Behavior Research Methods, 47(4):966–976. find
  16. Justine T. Kao, Jean Y. Wu, Leon Bergen, and Noah D. Goodman. 2014. Nonliteral understanding of number words. Proceedings of the National Academy of Sciences, 111(33):12002–12007. find our card
  17. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3128–3137. find our card
  18. Kazuya Kawakami, Chris Dyer, Bryan Routledge, and Noah A. Smith. 2016. Character sequence models for colorful words. In Proceedings of the 2016 Conference on Empirical Methods on Natural Language Processing (EMNLP), pages 1949–1954. find
  19. Diederik P. Kingma and Jimmy Lei Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  20. Ruud Koolen, Albert Gatt, Martijn Goudbeek, and Emiel Krahmer. 2011. Factors causing overspecifica-tion in definite descriptions. Journal of Pragmatics, 43(13):3231–3250. find
  21. Robert M. Krauss and Sidney Weinheimer. 1964. Changes in reference phrases as a function of frequency of usage in social interaction: A preliminary study. Psychonomic Science, 1(1–12):113–114. find
  22. David Lewis. 1979. Scorekeeping in a language game. Journal of Philosophical Logic, 8(1):339–359. find
  23. Brian McMahan and Matthew Stone. 2015. A Bayesian model of grounded color semantics. Transactions of the Association for Computational Linguistics, 3:103– 115. find
  24. Timothy Meo, Brian McMahan, and Matthew Stone. 2014. Generating and resolving vague color references. In Proceedings of the 18th Workshop on the Semantics and Pragmatics of Dialogue (SemDial), pages 107–115. find
  25. Will Monroe and Christopher Potts. 2015. Learning in the Rational Speech Acts model. In Proceedings of the 20th Amsterdam Colloquium, pages 1–12. find
  26. Will Monroe, Noah D. Goodman, and Christopher Potts. 2016. Learning to generate compositional color descriptions. In Proceedings of the 2016 Conference on Empirical Methods on Natural Language Processing (EMNLP), pages 2243–2248. find
  27. Randall Munroe. 2010. Color survey results. Online at http://blog.xkcd.com/2010/05/03/color-survey-results. ↗
  28. Sebastian Padó, 2006. User’s Guide to sigf: Significance Testing by Approximate Randomisation. http://www.nlpado.de/˜sebastian/ software/sigf.shtml. ↗
  29. Maike Paetzel, David Nicolas Racca, and David De-Vault. 2014. A multimodal corpus of rapid dialogue games. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC), pages 4189–4195. find
  30. Craige Roberts. 1996. Information structure in discourse: Towards an integrated formal theory of pragmatics. Working Papers in Linguistics—Ohio State University Department of Linguistics, pages 91–136. find our card
  31. Seymour Rosenberg and Bertram D. Cohen. 1964. Speakers’ and listeners’ processes in a word communication task. Science, 145(3637):1201–1203. find
  32. Gaurav Sharma, Wencheng Wu, and Edul N. Dalal. 2005. The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations. Color Research & Application, 30(1):21–30. find
  33. Nathaniel J. Smith, Noah D. Goodman, and Michael C. Frank. 2013. Learning and using language via recursive pragmatic reasoning about other agents. In Advances in Neural Information Processing Systems 26 (NIPS 2013), pages 3039–3047. find
  34. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27 (NIPS 2014), pages 3104–3112. find
  35. Michael K. Tanenhaus and Sarah Brown-Schmidt. 2008. Language processing in the natural world. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 363(1493):1105–1122. find
  36. Stefanie Tellex, Ross A. Knepper, Adrian Li, Daniela Rus, and Nicholas Roy. 2014. Asking for help using inverse semantics. In Robotics: Science and Systems. find
  37. Joshua B. Tenenbaum, Charles Kemp, Thomas L. Griffiths, and Noah D. Goodman. 2011. How to grow a mind: Statistics, structure, and abstraction. Science, 331(6022):1279–1285. find
  38. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the Human Language Technology Conference/North American Chapter of the Association for Computational Linguistics (HLT-NAACL), pages 173– 180. find
  39. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3156–3164. find our card
  40. Adam Vogel, Christopher Potts, and Dan Jurafsky. 2013. Implicatures and nested beliefs in approximate Decentralized-POMDPs. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL), pages 74–80. find
  41. Matthew D. Zeiler. 2012. ADADELTA: An adaptive learning rate method. arXiv preprint arXiv:1212.5701. ↗ our card