Elamur Research⌕
☰
home / papers / 2024 / Communicating with Speakers and Listeners of Different Pragmatic Levels

Communicating with Speakers and Listeners of Different Pragmatic Levels

Paper · 2024 · Kata Naszadi, Frans A Oliehoek and Christof Monz

Summary

The paper asks how communication and word learning are affected when speakers and listeners reason about each other to different depths. The authors use a Lewis-style reference game built on ShapeWorld. Speakers are exact Rational Speech Act (RSA) speakers with access to the true lexicon: S1 models a literal listener, and S3 models a pragmatic one. Learners are literal (L0) or pragmatic (L2) listeners. Each learner's lexicon is a CNN–GRU similarity score between an image and a message. For pragmatic learners, gradients flow through the whole recursive RSA computation during training, not just at test time.

Two main findings emerge. First, every learner learns better from the verbose, literal S1 than from the terse S3. Second, a pragmatic learner largely cancels the disadvantage of learning from concise pragmatic speech: an L0 trained with S1 performs about as well as an L2 trained with S3. This matters for systems that learn meaning from naturally under-specified human language. The lesson is that the learner's model of the speaker belongs inside training, not only inside inference.

Key points

  • Environment. The game is built on ShapeWorld and uses 6 colors and 5 shapes. Objects are placed on a 64x64 grid with random position, size and rotation. Target/distractor tuples are drawn from Dirichlet-sampled priors and co-occurrence conditionals. These come in two settings: near-uniform (Corr = 0) and correlated (Corr = 1).
  • Test data. There are 10 resampled distribution constellations with 3200 games each. The seed is fixed so every listener sees identical games.
  • Message length. S3 sends shorter messages than S1: 1.01–1.09 words versus 1.07–1.24 words across 2–4 distractors. In both cases, length grows with the number of distractors.
  • Evaluation-time upgrade (Table 2). An L0-trained lexicon was upgraded to L0 or L2 at test time. With S1 it scored 85.5/85.6. With S3, the L2 version did significantly better than L0 (81.2 vs 80.5).
  • Training-time speaker (Table 3). Learning from S1 beats learning from S3 for both L0 (80.7 vs 79.1) and L2 (84.8 vs 83.2). Significance was tested with Fisher's exact test. The setup used cost 0.6, N = 5 and Corr = 1. Code is in the naszka/rsa_backward repository.

Interesting ideas

  • I1. Learn through the reasoning, not just under it. The lexicon is trained by backpropagating through the recursive speaker/listener computation, so learned word meanings already account for the fact that observed utterances were chosen pragmatically.
  • I2. Simulated speakers with controlled depth as data generators. Using exact rational speakers of chosen depth produces data where pragmatic sophistication is a controlled variable rather than noise.
  • I3. Pick the alternative the speaker did not say. The listener reasons counterfactually: had the speaker meant the other red object, "square" would have been the more informative word.
  • I4. Explicit input first, concise input later. Every learner benefited from verbose, literal descriptions, which suggests a curriculum ordered from fully specified to under-specified language.
  • I5. Use distractor sets to stress pragmatics. Correlating the features of distractors raises the chance that literal and pragmatic messages diverge, which makes the effect measurable.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When training lexical or frame-scoring components on human-written text, the implicature layer could be placed inside the training objective. This keeps terse, implied usages from being read as evidence of weaker literal meanings.
  • I2 (learn): Self-play could generate questions or descriptions from both a literal teacher and a pragmatic teacher. The paired outputs then serve as controls that reveal whether a reader truly handles implicature.
  • I3 (frames): The prag crate could compute scalar and referential implicatures by enumerating the cheap alternative utterances available in the current world. It would then keep the reading under which the actual utterance was the best choice, producing an explainable trace.
  • I4 (learn): Silver training data could be staged so that early rounds use explicit, fully specified sentences. Elliptical and implicature-heavy text would be introduced only once the base lexicon is stable.
  • I5 (answer): Reference-resolution test items could be built with deliberately similar candidate entities, such as two foxes that differ in a single attribute. This would test whether the reader resolves under-specified mentions pragmatically.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations🎓Delft University of TechnologyDelft · Netherlands6 works · 290 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Kata Naszadi, Frans A Oliehoek and Christof Monz. 2024. Communicating with Speakers and Listeners of Different Pragmatic Levels. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.1213 · arXiv 2410.05851

@inproceedings{naszadi2024communicating,
  title = {{Communicating with Speakers and Listeners of Different Pragmatic Levels}},
  author = {Kata Naszadi, Frans A Oliehoek and Christof Monz},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1213},
  eprint = {2410.05851},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.1213}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. ↗
  2. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Meth-Robert D Hawkins, Michael Franke, Michael C Frank, Adele E Goldberg, Kenny Smith, Thomas L Griffiths, and Noah D Goodman. 2023. From partners to populations: A hierarchical bayesian account of coordination and convention. Psychological Review, 130(4):977.
  3. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). ↗
  4. Laurence Horn. 1984. Toward a new taxonomy for pragmatic inference: Q-based and r-based implicature. In Meaning, form, and use in context: Linguistic applications, pages 11–42. Georgetown University Press. find
  5. William S Horton and Richard J Gerrig. 2002. Speakers’ experiences and audience design: Knowing when and knowing how to adjust utterances to addressees. Journal of Memory and Language, 47(4):589–606. ↗
  6. Alexander Kuhnle and Ann A. Copestake. 2017. Shapeworld - a new test methodology for multimodal language understanding. ArXiv, abs/1704.04517. ↗
  7. Angeliki Lazaridou, Anna Potapenko, and Olivier Tiele-man. 2020. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7663–7674, Online. Association for Computational Linguistics. find
  8. Sang-Woo Lee, Yu-Jung Heo, and Byoung-Tak Zhang. 2018. Answerer in questioner’s mind: Information theoretic approach to goal-oriented visual dialog. Advances in neural information processing systems, 31. ↗
  9. David Kellogg Lewis. 1969. Convention: A Philosophical Study. Cambridge, MA, USA: Wiley-Blackwell. find
  10. Andy Liu, Hao Zhu, Emmy Liu, Yonatan Bisk, and Graham Neubig. 2023. Computational language acquisition with theory of mind. In The Eleventh International Conference on Learning Representations. ↗
  11. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. In International Conference on Learning Representations. ↗
  12. Jia E Loy and Vera Demberg. 2023. Perspective taking reflects beliefs about partner sophistication: Modern computer partners versus basic computer and human partners. Cognitive Science, 47(12):e13385. ↗
  13. Alexandra Mayn, JE Loy, and Vera Demberg. 2023. Individual differences in overspecification: reasoning and verbal fluency. ↗
  14. Alexandra Mayn, Jia E Loy, and Vera Demberg. 2024. Beliefs about the speaker’s reasoning ability influence pragmatic interpretation: Children and adults as speakers. ↗
  15. Bill McDowell and Noah Goodman. 2019. Learning from omission. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 619–628, Florence, Italy. Association for Computational Linguistics. find
  16. Will Monroe and Christopher Potts. 2015. ing in the rational speech acts model. abs/1510.06807. ↗
  17. Learn-CoRR, Minh Thi Thuy Nguyen. 2022. Interlanguage pragmatics as communicative competence. chapter 8, pages 135–151. Taylor & Francis. find
  18. Julia White, Jesse Mu, and Noah D. Goodman. 2020. Learning to refer informatively by amortizing pragmatic reasoning. In Proceedings of the 42th Annual Meeting of the Cognitive Science Society - Developing a Mind: Learning in Humans, Animals, and Machines, CogSci 2020, virtual, July 29 - August 1, 2020. cognitivesciencesociety.org. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/naszadi-2024-communicating-speakers-listeners-different-pragmatic/