Elamur Research⌕
☰
home / papers / 2024 / Are LLMs good pragmatic speakers?

Are LLMs good pragmatic speakers?

Paper · 2024 · Mingyue Jian and N. Siddharth

Summary

The paper asks whether a large language model, used as a speaker, prefers descriptions that are informative in context rather than merely true. Most earlier work tested LLMs as listeners. The authors take the Rational Speech Act (RSA) model as a reference for pragmatic behaviour and build reference games from the furniture domain of the TUNA corpus. In each game, seven objects are described by size, colour, type and orientation. Candidate descriptions of a target come from two sources: beam-search outputs of Llama3-8B-Instruct, and an exhaustive set built by rule from feature combinations. Each candidate gets two scores: its sequence probability under the LLM, and its RSA speaker probability. RSA is computed with two meaning functions, one prompt-based and one rule-based. The two scores agree only moderately. The LLM tends to favour literal, over-complete descriptions and largely ignores the pragmatic ones. The authors therefore find no convincing evidence that the model behaves pragmatically. The setup is a reusable recipe for probing speaker-side pragmatics.

Key points

  • The data consist of 2,940 TUNA reference games with 386,510 scored utterances: 88,310 from top-k beam search and 298,200 from logical construction.
  • The two meaning functions were checked against human labels:
    • The rule-based one, which rejects an utterance if any word contradicts an object feature, reached about 99.9% accuracy.
    • The 3-shot prompt-based one, which uses P(Yes) normalised over Yes/No, reached 97.3%.
  • Correlations between LLM and RSA scores within each object's candidate set:
    • With the rule-based function: PCC 0.303, SRCC 0.736.
    • With the prompt-based function: PCC 0.291, SRCC 0.606.
    • Agreement is higher for the rule-built candidates than for the beam-search ones.
  • Changing the rationality parameter α (0.2 to 3.0) makes the RSA distribution sharper but does not change the weak overall alignment.
  • Within the top-k group alone, rank correlation falls to roughly −0.25…0.50. The LLM separates its own outputs from constructed ones, but it does not order its own outputs pragmatically.

Interesting ideas

  • I1. Instead of judging free generations, score a fixed, enumerated set of alternative utterances with both the model and an explicit pragmatic reference, then compare the rankings with Pearson and Spearman correlation; this makes "is it pragmatic?" a measurable question.
  • I2. A literal meaning function can be built purely from feature exclusion: an utterance fits an object unless one of its words names a feature the object lacks. This simple rule beat the LLM judge in accuracy.
  • I3. The space of utterances is built logically as every subset of attribute values, with omitted heads replaced by "thing". This yields both literal and pragmatic variants in a clean noun-phrase form.
  • I4. The paper checks its meaning functions against human labels before trusting them as the reference. A metric is only as good as its literal-truth oracle.
  • I5. RSA charges a cost proportional to utterance length, which ties brevity to Gricean quantity. Informativity divided by length gives a cheap preference for minimal sufficient descriptions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova's evaluation could rank candidate answers or descriptions with both a learned scorer and a symbolic RSA scorer, using their disagreement as a diagnostic signal.
  • I2 (frames): the prag crate could implement RSA-style implicature over entity features from the text's world, with this exclusion rule as the transparent literal listener.
  • I3 (back): generation could enumerate referring expressions for an entity this way and keep the shortest one that singles it out, giving explainable referring-expression choice.
  • I4 (learn): when an LLM teacher tier acts as a truth judge, compare it with a rule-based judge on a labelled sample first, as a control.
  • I5 (answer): readers that answer "which one?" questions could rank candidate referents by the RSA pragmatic-listener score rather than by literal match alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Mingyue Jian and N. Siddharth. 2024. Are LLMs good pragmatic speakers?. arXiv:2411.01562.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2411.01562 · arXiv 2411.01562

@misc{jian2024are,
  title = {{Are LLMs good pragmatic speakers?}},
  author = {Mingyue Jian and N. Siddharth},
  year = {2024},
  doi = {10.48550/arxiv.2411.01562},
  eprint = {2411.01562},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.01562}
}

References

28 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. ↗
  2. J. Andreas and D. Klein. Reasoning about pragmatics with neural listeners and speakers. arXiv preprint arXiv:1604.00562, 2016. ↗
  3. bartowski. bartowski/meta-llama-3.1-8b-instruct-gguf, 2024. URL https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF. Accessed: 2024-10-29. ↗
  4. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  5. G. Carenini, L. Bodot, L. Bischetti, W. Schaeken, and V. Bambini. Large language models behave (almost) as rational speech actors: Insights from metaphor understanding. In NeurIPS 2023 workshop: Information-Theoretic Principles in Cognitive Systems, 2023. find
  6. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel. Palm: Scaling language modeling with pathways, 2022. ↗
  7. J. Degen. The rational speech act framework. Annual Review of Linguistics, 9(1):519–540, 2023. ↗
  8. D. C. Dennett. True Believers:The Intentional Strategy and Why It Works. In Mind Design II: Philosophy, Psychology, and Artificial Intelligence. The MIT Press, 03 1997. ISBN 9780262275071. doi: 10.7551/mitpress/4626.003.0003. URL https://doi.org/10.7551/ mitpress/4626.003.0003. ↗
  9. M. C. Frank and N. D. Goodman. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998, 2012. doi: 10.1126/science.1218633. URL https://www.science.org/doi/abs/10.1126/science.1218633. ↗
  10. N. D. Goodman and M. C. Frank. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20(11):818–829, 2016. ↗
  11. H. P. Grice. Logic and conversation. In Speech acts, pages 41–58. Brill, 1975. ↗
  12. J. Hu, S. Floyd, O. Jouravlev, E. Fedorenko, and E. Gibson. A fine-grained comparison of pragmatic language understanding in humans and language models. arxiv. arXiv preprint arXiv:2212.06801, 2022. ↗
  13. R. M. Krauss and S. Weinheimer. Changes in reference phrases as a function of frequency of usage in social interaction: A preliminary study. Psychonomic Science, 1:113–114, 1964. ↗
  14. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, 2019.
  15. O. Lieber, O. Sharir, B. Lenz, and Y. Shoham. Jurassic-1: Technical details and evaluation. White Paper. AI21 Labs, 1, 2021. find
  16. A. Louis, D. Roth, and F. Radlinski. " i’d rather just go to bed": Understanding indirect answers. arXiv preprint arXiv:2010.03450, 2020. ↗
  17. K. Mahowald, A. A. Ivanova, I. A. Blank, N. Kanwisher, J. B. Tenenbaum, and E. Fedorenko. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 2024. ↗
  18. W. Monroe, R. X. Hawkins, N. D. Goodman, and C. Potts. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5:325–338, 2017. ↗
  19. A. Nematzadeh, K. Burns, E. Grant, A. Gopnik, and T. L. Griffiths. Evaluating theory of mind in question answering. arXiv preprint arXiv:1808.09352, 2018. ↗
  20. K. Nguyen. Language models are bounded pragmatic speakers: Understanding rlhf from a bayesian cognitive modeling perspective. arXiv preprint arXiv:2305.17760, 2023. ↗
  21. D. Premack and G. Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515–526, 1978. ↗
  22. S. Rosenberg and B. D. Cohen. Speakers’ and listeners’ processes in a word-communication task. Science, 145(3637):1201–1203, 1964. ↗
  23. L. Ruis, A. Khan, S. Biderman, S. Hooker, T. Rocktäschel, and E. Grefenstette. The goldilocks of pragmatic understanding: Fine-tuning strategy matters for implicature resolution by llms. Advances in Neural Information Processing Systems, 36, 2024.
  24. S. L. Sravanthi, M. Doshi, T. P. Kalyan, R. Murthy, P. Bhattacharyya, and R. Dabre. Pub: A pragmatics understanding benchmark for assessing llms’ pragmatics capabilities. arXiv preprint arXiv:2401.07078, 2024. ↗
  25. R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022. ↗
  26. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. ↗
  27. K. van Deemter, I. van der Sluis, and A. Gatt. Building a semantically transparent corpus for the generation of referring expressions. In Proceedings of the Fourth International Natural Language Generation Conference, pages 130–132, 2006. find
  28. J. White, J. Mu, and N. D. Goodman. Learning to refer informatively by amortizing pragmatic reasoning. arXiv preprint arXiv:2006.00418, 2020. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jian-2024-llms-good-pragmatic-speakers/