Elamur Research⌕
☰
home / papers / 2023 / Evaluating statistical language models as pragmatic reasoners

Evaluating statistical language models as pragmatic reasoners

Summary

Vague words such as "strong" have no fixed cutoff, so their interpretation depends on context. Classic accounts compute this with recursive Bayesian models such as Rational Speech Acts (RSA). Lipkin and colleagues test whether a large language model can do this work without explicit computation. They describe a tug-of-war world as a Church probabilistic program in which player strength has a normal prior. The LLM then translates English statements about a player named Jack into conditioning statements of the form strength > θ. Scoring candidate programs across θ values yields a full threshold distribution in a single pass. The authors compare these distributions with estimates from 60 human participants in two experiments: one on qualifiers, negation and polarity inversion, and one on comparison classes. The model matches people well for positive descriptions and class-relative phrases, even when the class is named only through a synonym or allusion. It diverges for "weak", "not weak" and "not strong". The findings bear on using LLMs as semantic parsers for non-literal meaning.

Key points

  • Setup: The domain is a tug-of-war world written in Church, with a strength prior of N(50, 20).
  • Scoring: With the threshold θ stepping through values between 0 and 100 in increments of 10, the LLM assigns a score to each corresponding program (condition (> (strength 'jack) θ)). A non-vague control sentence ("at least average strength") should concentrate probability near θ = 50.
  • Experiments: E1 covers qualifiers such as "pretty strong", plus negation and the reversed-polarity "weak" materials. E2 covers comparison classes such as "strong for a beginner", stated as the exact class, as synonyms, or as allusions. Similarity between model and human distributions was tested at p < 0.05 with FDR correction.
  • Findings: The model handled positive and class-conditioned phrases in a human-like way and was robust to distracting references in the context. It failed to compose meaning with negation and polarity inversion, which is consistent with earlier reports of LLM trouble with negation.
  • Resources and future work: Code is released in the probsem GitHub repository (CogSci2023 branch). The authors propose executing LLM-inferred programs in a probabilistic programming interpreter and combining LLM estimates with RSA or SMC-style inference for harder cases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing · AI & machine learning

Cited by

13 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 12 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.01020 · arXiv 2305.01020

@misc{lipkin2023evaluating,
  title = {{Evaluating statistical language models as pragmatic reasoners}},
  author = {Lipkin, Benjamin and Wong, Lionel and Grand, Gabriel and Tenenbaum, Joshua B},
  year = {2023},
  doi = {10.48550/arxiv.2305.01020},
  eprint = {2305.01020},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.01020}
}

References

53 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Acquaviva, S., Pu, Y., Kryven, M., Sechopoulos, T., Wong, C., Ecanow, G. E., . . . Tenenbaum, J. B. (2021). Communicating natural programs to humans and machines. arXiv preprint arXiv:2106.07824. ↗
  2. Artzi, Y., Das, D., & Petrov, S. (2014). Learning compact lexicons for ccg semantic parsing. find
  3. Austin, J. L. (1975). How to do things with words. Oxford university press. ↗
  4. Barner, D., & Snedeker, J. (2008). Compositionality and statistics in adjective acquisition: 4-year-olds interpret tall and short based on the size distributions of novel noun referents. Child development, 79(3), 594–608. ↗
  5. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1), 289–300. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., . . . others (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. ↗ our card
  6. Brewer, W. F., & Stone, J. B. (1975). Acquisition of spatial antonym pairs. Journal of Experimental Child Psychology, 19(2), 299–307. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., . . . others (2021). Evaluating large language models trained on code. arXiv. ↗ our card
  7. Clark, H. H. (1996). Using language. Cambridge university press. find
  8. Collins, K. M., Wong, C., Feng, J., Wei, M., & Tenenbaum, J. B. (2022). Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks. arXiv preprint arXiv:2205.05718. ↗
  9. Cooper, R., Dobnik, S., Lappin, S., & Larsson, S. (2015). Probabilistic type theory and natural language semantics. Linguistic issues in language technology, 10. ↗
  10. Cresswell, M. J. (1976). The semantics of degree. In Montague grammar (pp. 261–292). Elsevier. ↗
  11. Creswell, A., Shanahan, M., & Higgins, I. (2022). Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712. ↗
  12. Dasgupta, I., & Gershman, S. J. (2021). Memory as a computational resource. Trends in Cognitive Sciences, 25(3), 240–251. ↗
  13. Frank, M., & Goodman, N. D. (2012). Predicting pragmatic reasoning in language games. Science, 336(6084), 998– 998. ↗ our card
  14. Fried, D., Tomlin, N., Hu, J., Patel, R., & Nematzadeh, A. (2022). Pragmatics in grounded language learning: Phenomena, tasks, and modeling approaches. arXiv preprint arXiv:2211.08371. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., . . . Neubig, G. (2022). Pal: Program-aided language models. arXiv. ↗ our card
  15. Gershman, S., & Goodman, N. (2014). Amortized inference in probabilistic reasoning. In Proceedings of the annual meeting of the cognitive science society (Vol. 36). find
  16. Gershman, S., Horvitz, E. J., & Tenenbaum, J. B. (2015). Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science, 349(6245), 273–278. ↗
  17. Gerstenberg, T., & Goodman, N. D. (2012). Ping pong in church: Productive use of concepts in human probabilistic inference. In Proceedings of the annual meeting of the cognitive science society (Vol. 34). ↗
  18. Goodman, N. D., & Frank, M. C. (2016). Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20(11), 818–829. ↗
  19. Goodman, N. D., & Lassiter, D. (2015). Probabilistic semantics and pragmatics: Uncertainty in language and thought. The handbook of contemporary semantic theory, 2nd edition. Wiley-Blackwell. ↗
  20. Goodman, N. D., Mansinghka, V., Roy, D., Bonawitz, K., & Tenenbaum, J. B. (2012). Church: a language for generative models. arXiv. ↗
  21. Goodman, N. D., & Stuhlmüller, A. (2013). Knowledge and implicature: Modeling language understanding as social cognition. Topics in cognitive science, 5(1), 173–184. ↗
  22. Goodman, N. D., & Tenenbaum, J. B. (2010). Probabilistic Models of Cognition (First ed.). http://v1.probmods.org/. ↗
  23. Goodman, N. D., Tenenbaum, J. B., & Gerstenberg, T. (2014). Concepts in a probabilistic language of thought. Center for Brains, Minds and Machines (CBMM) Memos, 010. ↗
  24. Grice, P. (1989). Studies in the way of words. Harvard University Press. find
  25. Hosseini, A., Reddy, S., Bahdanau, D., Hjelm, R. D., Sordoni, A., & Courville, A. (2021). Understanding by understanding not: Modeling negation in language models. arXiv preprint arXiv:2105.03519. ↗
  26. Hu, J., Floyd, S., Jouravlev, O., Fedorenko, E., & Gibson, E. (2022). A fine-grained comparison of pragmatic language understanding in humans and language models. arXiv preprint arXiv:2212.06801. ↗ our card
  27. Hu, J., Levy, R., Degen, J., & Schuster, S. (2023). Expectations over unspoken alternatives predict pragmatic inferences. arXiv preprint arXiv:2304.04758. ↗ our card
  28. Kassner, N., & Schütze, H. (2019). Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. arXiv preprint arXiv:1911.03343. ↗ our card
  29. Kennedy, C. (2007). Vagueness and grammar: The semantics of relative and absolute gradable adjectives. Linguistics and philosophy, 30(1), 1–45. ↗
  30. Klatzky, R. L., Clark, E. V., & Macken, M. (1973). Asymmetries in the acquisition of polar adjectives: linguistic or conceptual? Journal of Experimental Child Psychology, 16(1), 32–46. ↗
  31. Klein, E. (1980). A semantics for positive and comparative adjectives. Linguistics and philosophy, 4(1), 1–45. ↗
  32. Kratzer, A., & Irene, H. (1998). Semantics in generative grammar (Vol. 1185). Blackwell Oxford. find
  33. Kripke, S. A. (1963). Semantical analysis of modal logic i normal modal propositional calculi. Mathematical Logic Quarterly, 9(5-6), 67–96. ↗
  34. Kwiatkowski, T., Zettlemoyer, L., Goldwater, S., & Steedman, M. (2011). Lexical generalization in ccg grammar induction for semantic parsing. In Proceedings of the 2011 conference on empirical methods in natural language processing (pp. 1512–1523). find
  35. Lassiter, D., & Goodman, N. D. (2017). Adjectival vagueness in a bayesian model of interpretation. Synthese, 194(10), 3801–3836. find
  36. Levinson, S. C. (1983). Pragmatics. Cambridge university press. find
  37. Lin, J. (1991). Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1), 145–151. Mishra, S., Finlayson, M., Lu, P., Tang, L., Welleck, S., Baral, C., . . . others (2022). Lila: A unified benchmark for mathematical reasoning. arXiv preprint arXiv:2210.17517. ↗ our card
  38. Montague, R. (1973). The proper treatment of quantification in ordinary english. In Approaches to natural language: Proceedings of the 1970 stanford workshop on grammar and semantics (pp. 221–242). ↗
  39. Nelder, J. A., & Mead, R. (1965). A simplex method for function minimization. The computer journal, 7(4), 308– 313. ↗
  40. Partee, B. B., ter Meulen, A., & Wall, R. (1990). Mathematical methods in linguistics (Vol. 30). Springer Science & Business Media. find
  41. Piantadosi, S. T. (2023). Modern language models refute chomsky’s approach to language. Lingbuzz Preprint, lingbuzz/007180. ↗
  42. Qing, C., & Franke, M. (2014). Gradable adjectives, vagueness, and optimal language use: A speaker-oriented model. In Semantics and linguistic theory (Vol. 24, pp. 23–41). ↗
  43. Ruis, L., Khan, A., Biderman, S., Hooker, S., Rocktäschel, T., & Grefenstette, E. (2022). Large language models are not zero-shot communicators. arXiv preprint arXiv:2210.14986. ↗
  44. Searle, J. R. (1969). Speech acts: An essay in the philosophy of language (Vol. 626). Cambridge university press. ↗ our card
  45. Steedman, M. (2001). The syntactic process. MIT press. ↗
  46. Tenney, I., Das, D., & Pavlick, E. (2019). Bert rediscovers the classical nlp pipeline. arXiv preprint arXiv:1905.05950. ↗ our card
  47. Tessler, M. H., & Goodman, N. D. (2022). Warm (for winter): Inferring comparison classes in communication. Cognitive Science, 46(3), e13095. ↗
  48. Tessler, M. H., Tsvilodub, P., Snedeker, J., & Levy, R. P. (2020). Informational goals, sentence structure, and comparison class inference. In Proceedings of the annual conference of the cognitive science society. ↗
  49. Van Eijck, J., & Lappin, S. (2012). Probabilistic semantics for natural language. Logic and interactive rationality (LIRA), 2, 17–35. find
  50. White, J., Mu, J., & Goodman, N. D. (2020). Learning to refer informatively by amortizing pragmatic reasoning. arXiv preprint arXiv:2006.00418. ↗
  51. Wittgenstein, L. (1953). Philosophical investigations. Wong, L., Grand, G., Lew, A., Andreas, J., Goodman, N. D., Mansinghka, V., & Tenenbaum, J. B. (prep.). Translating from natural language to the language of thought. preprint. find
  52. Wu, M., & Goodman, N. (2022). Foundation posteriors for approximate probabilistic inference. arXiv preprint arXiv:2205.09735. ↗
  53. Zelikman, E., Huang, Q., Poesia, G., Goodman, N. D., & Haber, N. (2022). Parsel: A unified natural language framework for algorithmic reasoning. arXiv preprint arXiv:2212.10561. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lipkin-2023-evaluating-statistical-language-models-pragmatic/