Elamur Research⌕
☰
home / papers / 2022 / Color Overmodification Emerges from Data-Driven Learning and Pragmatic Reasoning

Color Overmodification Emerges from Data-Driven Learning and Pragmatic Reasoning

Paper · 2022 · Fei Fang, Kunal Sinha, Noah D. Goodman et al.

Summary

Human speakers often name an object's color even when its shape alone would single it out. This redundancy is common for objects that come in many colors. It drops when the color is the expected one, and it is thought to depend on how visually salient color is. Earlier accounts, including RSA-based ones, start from a ready-made semantics, so they cannot explain where that semantics comes from. The authors train neural agents directly on images and utterances from ShapeWorld reference games, in which one target is shown among three pictures. They compare a literal speaker with an RSA speaker that reasons about an internal learned listener. The network architecture stays fixed; only the training world changes: a uniform baseline, a world where shapes tend to have typical colors, and a world where color is hard to perceive. Neither speaker ever sees overmodification in training. The RSA speaker still reproduces human-like trends: more color mention for atypical colors and less when color is hard to extract. The literal speaker produces too little color mention and, under low salience, moves in the opposite direction. The paper argues that pragmatic reasoning and data statistics together can account for the phenomenon.

Key points

  • Data: ShapeWorld games built from 6 colors × 4 shapes (24 combinations), each with three images on a black background. There are four context conditions, and each game is paired with the shortest uniquely identifying utterance from an 11-word vocabulary (six colors, four shapes, and "shape").
  • Listener semantics: a sigmoid over the dot product of an image-encoder vector and an utterance-encoder vector (d = 1024). The evaluation listener picks a referent with probability proportional to the exponent of this score, and it is trained on non-contextual data.
  • Exp. 2 (TYPICALITY vs UNIFORM): the internal listener learns that a bare "circle" fits red circles better than circles of other colors (interaction β = −.76, p < .001). The RSA speaker's color rates come close to the human figures from Degen et al. (2020), about 25% for typical and 55% for atypical colors, while the literal speaker underpredicts them.
  • Exp. 3 (LOW-SALIENCE): each object's color is carried by a single pixel. The RSA speaker overmodifies less and the literal speaker more (speaker × salience interaction β = −.20). The learned uncertainty for color words rises roughly tenfold, from 0.008 to 0.094.
  • Salience is operationalized simply as ease of feature extraction.

Interesting ideas

  • I1. Overinformative wording falls out of graded, learned word–object fit combined with listener modelling, with no explicit "add redundant adjective" rule; a hearer can also reverse this, reading a redundant modifier as a hint that the property is unusual.
  • I2. Hold the model fixed and change one statistic of a synthetic training world, so that differences in behavior can be attributed causally to that statistic.
  • I3. The authors explain the speaker's choices by inspecting the graded semantic values of its internal listener, which turns an opaque network decision into readable fit scores.
  • I4. A referring expression is chosen by asking how a simulated listener with uncertain semantics would interpret each candidate, rather than by applying a fixed minimal-description algorithm.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): implicature in prag could flag an unnecessary color or attribute adjective in a definite description as a weak signal that the property is atypical for that kind of object.
  • I2 (learn): build generated mini-corpora that differ in a single distributional factor, such as how often a modifier appears, and use them as negative controls to check which behavior is really driven by the data.
  • I3 (world): keep learned typicality scores, such as how well a bare noun matches each property value, so the world model can fill in default states (an unmodified "banana" is yellow) with an explainable confidence.
  • I4 (back): when generating text from the meaning graph, rank candidate noun phrases with an internal reader model and add a modifier when that reader is uncertain, for example about an atypical color.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Fei Fang, Kunal Sinha, Noah D. Goodman, Christopher Potts and Elisa Kreiss. 2022. Color Overmodification Emerges from Data-Driven Learning and Pragmatic Reasoning. arXiv:2205.09172.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2205.09172 · arXiv 2205.09172

@misc{fang2022color,
  title = {{Color Overmodification Emerges from Data-Driven Learning and Pragmatic Reasoning}},
  author = {Fei Fang, Kunal Sinha, Noah D. Goodman, Christopher Potts and Elisa Kreiss},
  year = {2022},
  doi = {10.48550/arxiv.2205.09172},
  eprint = {2205.09172},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.09172}
}

References

49 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andreas, J., & Klein, D. (2016). Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (pp. 1173–1182). Austin, Texas: Association for Computational Linguistics. doi: 10.18653/v1/D16-1125 ↗
  2. Arnold, J. E. (2008). Reference production: Production-internal and addressee-oriented processes. Language and cognitive processes, 23(4), 495–527. ↗
  3. Brown-Schmidt, S., & Konopka, A. E. (2011). Experimental approaches to referential domains and the on-line processing of referring expressions in unscripted conversation. Information, 2(2), 302–326. ↗
  4. Cho, K., van Merriënboer, B., Gu̇lçehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN Encoder-Decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 1724–1734). ↗
  5. Cohn-Gordon, R., Goodman, N. D., & Potts, C. (2018). Pragmatically informative image captioning with character-level inference. In Proceedings of the 2018 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies (pp. 439–443). Stroudsburg, PA: Association for Computational Linguistics.
  6. Cohn-Gordon, R., Goodman, N. D., & Potts, C. (2019). An incremental iterated response model of pragmatics. In Proceedings of the Society for Computation in Linguistics (SCiL) 2019 (pp. 81–90). ↗
  7. Dale, R., & Reiter, E. (1995). Computational interpretations of the Gricean maxims in the generation of referring expressions. Cognitive Science, 19(2), 233–263. ↗
  8. Davies, C., & Katsos, N. (2013). Are speakers and listeners ‘only moderately Gricean’? An empirical response to Engelhardt et al. (2006). Journal of Pragmatics, 49(1), 78– 106. find
  9. Degen, J., Hawkins, R. D., Graf, C., Kreiss, E., & Goodman, N. D. (2020). When redundancy is useful: A Bayesian approach to “overinformative” referring expressions. Psychological Review, 127(4), 591. find
  10. Engelhardt, P. E., Bailey, K. G., & Ferreira, F. (2006). Do speakers and listeners observe the Gricean Maxim of Quantity? Journal of Memory and Language, 54(4), 554–573. ↗
  11. Frank, M. C., & Goodman, N. D. (2012). Predicting pragmatic reasoning in language games. Science, 336(6084), 998–998. ↗
  12. Franke, M. (2009). Signal to act: Game theory in pragmatics. Institute for Logic, Language and Computation Amsterdam. find
  13. Fried, D., Andreas, J., & Klein, D. (2018). Unified pragmatic models for generating and following instructions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (pp. 1951–1963). New Orleans, Louisiana: Association for Computational Linguistics. ↗
  14. Gatt, A., van Gompel, R. P., Krahmer, E., & van Deemter, K. (2011). Non-deterministic attribute selection in reference production. In Proceedings of the CogSci workshop Production of Referring Expressions. find
  15. Golland, D., Liang, P., & Klein, D. (2010). A game-theoretic approach to generating spatial descriptions. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (pp. 410–419). find
  16. Goodman, N. D., & Frank, M. (2016). Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences, 20. ↗
  17. Goodman, N. D., & Stuhlmüller, A. (2013). Knowledge and implicature: Modeling language understanding as social cognition. Topics in Cognitive Science, 5(1), 173–184. ↗
  18. Hawkins, R., & Goodman, N. (2017). Why do you ask? The informational dynamics of questions and answers. PsyArXiv. find
  19. Heller, D. (2020). The production and comprehension of referring expressions: Definite description. Language and Linguistics Compass, 14(5), e12370. ↗
  20. Jager, G. (2007). Game dynamics connects semantics and pragmatics. Game Theory and Linguistic Meaning, 89– 102. find
  21. Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., & Girshick, R. (2017). Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2901–2910). ↗
  22. Kao, J. T., Wu, J. Y., Bergen, L., & Goodman, N. D. (2014). Nonliteral understanding of number words. Proceedings of the National Academy of Sciences, 111(33), 12002–12007. ↗
  23. Kennedy, C. (2007). Vagueness and grammar: The semantics of relative and absolute gradable adjectives. Linguistics and Philosophy, 30(1), 1–45. doi: 10.1007/s10988-006-9008-0 ↗
  24. Kingma, D. P., & Ba, J. (2015). Adam: A method for Stochastic Optimization. In Y. Bengio & Y. LeCun (Eds.), 3rd International Conference on Learning Representations. ↗
  25. Koolen, R., Gatt, A., Goudbeek, M., & Krahmer, E. (2011). Factors causing overspecification in definite descriptions. Journal of Pragmatics, 43(13), 3231–3250. ↗
  26. Kreiss, E., & Degen, J. (2020). Production expectations modulate contrastive inference. In Proceedings of the Annual Meeting of the Cognitive Science Society. find
  27. Kuhnle, A., & Copestake, A. A. (2017). Shapeworld - A new test methodology for multimodal language understanding. CoRR, abs/1704.04517. ↗
  28. LeCun, Y., & Bengio, Y. (1995). Convolutional networks for images, speech, and time series. In M. A. Arbib (Ed.), (p. 255-–258). find
  29. Lewis, D. K. (1969). Convention: A philosophical study. Wiley-Blackwell. find
  30. Long, M., Moore, I., Mollica, F., & Rubio-Fernandez, P. (2021). Contrast perception as a visual heuristic in the formulation of referential expressions. Cognition, 217, 1048-79. ↗
  31. Mitchell, M., Reiter, E., & Van Deemter, K. (2013). Typicality and object reference. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 35). find
  32. Monroe, W., Hawkins, R. X., Goodman, N. D., & Potts, C. (2017). Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5, 325–338. ↗
  33. Monroe, W., Hu, J., Jong, A., & Potts, C. (2018). Generating Bilingual Pragmatic Color References. In Proceedings of the 2018 Conference of the North American chapter of the Association for Computational Linguistics: Human language technologies (pp. 2155–2165). Stroudsburg, PA: Association for Computational Linguistics. find
  34. Nadig, A. S., & Sedivy, J. C. (2002). Evidence of perspective-taking constraints in children’s on-line reference resolution. Psychological Science, 13(4), 329–336. ↗
  35. Nie, A., Cohn-Gordon, R., & Potts, C. (2020). Pragmatic Issue-Sensitive Image Captioning. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 1924–1938). Online: Association for Computational Linguistics.
  36. Pechmann, T. (1989). Incremental speech production and referential overspecification. Linguistics, 27(1), 89–110. ↗
  37. Rubio-Fernández, P. (2016). How redundant are redundant color adjectives? An efficiency-based analysis of color overspecification. Frontiers in Psychology, 7, 153. find
  38. Rubio-Fernandez, P. (2021). Color discriminability makes over-specification efficient: Theoretical analysis and empirical evidence. Humanities and Social Sciences Communications, 8(1), 1–15. ↗
  39. Rubio-Fernandez, P., Terrasa, H. A., Shukla, V., & Jara-Ettinger, J. (2019). Contrastive inferences are sensitive to informativity expectations, adjective semantics and visual salience. PsyArXiv. ↗
  40. Sedivy, J. C. (2003). Pragmatic versus form-based accounts of referential contrast: Evidence for effects of informativity expectations. Journal of Psycholinguistic Research, 32(1), 3–23. ↗
  41. Sedivy, J. C. (2005). Evidence for Gricean Mechanisms in Online Language Interpretation. In Approaches to studying world-situated language use: Bridging the language-as-product and language-as-action traditions (p. 345). MIT press. find
  42. Syrett, K., Kennedy, C., & Lidz, J. (2009). Meaning and Context in Children’s Understanding of Gradable Adjectives. Journal of Semantics, 27(1), 1-35. ↗
  43. Tanaka, J. W., & Presnell, L. M. (1999). Color diagnosticity in object recognition. Perception & Psychophysics, 61, 1140–1153. ↗
  44. van Gompel, R. P., van Deemter, K., Gatt, A., Snoeren, R., & Krahmer, E. J. (2019). Conceptualization in reference production: Probabilistic modeling and experimental testing. Psychological Review, 126(3), 345. van Tiel, B., Franke, M., & Sauerland, U. (2021). Probabilistic pragmatics explains gradience and focality in natural language quantification. Proceedings of the National Academy of Sciences, 118(9). ↗
  45. Vedantam, R., Bengio, S., Murphy, K., Parikh, D., & Chechik, G. (2017). Context-aware captions from context-agnostic supervision. In CVPR. find
  46. Waldon, B., & Degen, J. (2021). Modeling cross-linguistic production of referring expressions. In Proceedings of the Society for Computation in Linguistics (Vol. 4). ↗
  47. Wang, R., White, J., Mu, J., & Goodman, N. (2021). Calibrate your listeners! Robust communication-based training for pragmatic speakers. In Findings of the Association for Computational Linguistics: EMNLP 2021 (pp. 977–984).
  48. Westerbeek, H., Koolen, R., & Maes, A. (2015). Stored object knowledge and the production of referring expressions: the case of color typicality. Frontiers in Psychology, 6, 935. ↗
  49. White, J., Mu, J., & Goodman, N. D. (2020). Learning to refer informatively by amortizing pragmatic reasoning. arXiv preprint arXiv:2006.00418. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fang-2022-color-overmodification-emerges-data-driven/