Elamur Research⌕
☰
home / papers / 2024 / Experimental Pragmatics with Machines: Testing LLM Predictions for the Inferences of Plain and Embedded Disjunctions

Experimental Pragmatics with Machines: Testing LLM Predictions for the Inferences of Plain and Embedded Disjunctions

Paper · 2024 · Polina Tsvilodub, Paul Marty, Sonia Ramotowska et al.

Summary

The paper asks whether large language models can add evidence to an open debate in pragmatics. The debate concerns the inferences triggered by "or": ignorance (II), distributive (DI) and free choice (FC). Are these ordinary scalar implicatures (SI)? And do their POSSIBILITY and DISTRIBUTIVE parts depend on first deriving UNCERTAINTY and NEGATED UNIVERSAL? The authors adapt the human "mystery box" paradigm for LLMs. Three boxes are visible, and a hidden fourth box matches one of them; a character's sentence is then rated good or bad. Materials come from eight human experiments (Marty, Romoli et al.; Degano et al.; Marty, Ramotowska et al.). The authors compare model acceptance rates with human rates in target contexts built to separate the inferences, plus true and false controls. The best models, GPT-3.5 and then L-70b, reproduce the large human gap between disjunction inferences and plain SIs. They also mostly show POS and DI arising without UNC and NU, which goes against the traditional implicature account. Fit is uneven, especially for modal sentences. The authors therefore propose criteria for when LLM evidence should count in linguistic theory.

Key points

  • The study weighs three accounts against each other: the traditional implicature account (TIA), Bar-Lev & Fox's (2020) modified version of the implicature view (RIA), and Aloni's (2022) non-implicature account (NIA). They differ on whether FC, II and DI are implicatures and whether POS and DI arise independently.
  • The human data come from three sources:
    • MRo (Exps 4–6), which pits FC, DI and II against SI with 9 test trials per inference.
    • D, which tests plain disjunction with TARGET-1 vs TARGET-2 contexts over 36 test trials.
    • MRa, which tests disjunction under nominal "every", epistemic "must" and deontic "must" over 24 test trials.
  • The models tested were GPT-3.5, L-70b, Mixtral (base and instruct) and Mistral-Instruct.
    • Mistral-Instruct accepted both target conditions almost every time.
    • L-70b fit the D data best but did poorly on FC.
    • GPT-3.5 explained the most variance overall.
  • The TARGET-1 vs TARGET-2 contrast for DI appeared in all models with nominal quantifiers. It was absent for GPT-3.5 and Mistral-Instruct with modals and weaker for the other models.
  • The authors report that model size preliminarily predicts fit to human data. They also flag sensitivity to prompt wording and to negation as open risks.

Interesting ideas

  • I1. The authors reuse a controlled human paradigm unchanged: a visual "mystery box" world turned into text, with a truth-value judgment, so that model and human numbers are directly comparable. This reuse of human experiments could benchmark any interpretation system.
  • I2. Target contexts are built so that a sentence comes out false only if a specific inference was drawn, making acceptance rate a proxy for how strongly that inference fires.
  • I3. Inferences are split into sub-parts (UNC vs POS, NU vs DI), and contexts are designed that make one part true and the other false, testing whether the parts are derived independently or in sequence.
  • I4. The authors argue that system output on one target condition means little without consistent behaviour across related phenomena, full response distributions, error analysis and control performance.
  • I5. The paper treats a model's results as more trustworthy evidence the more its performance tracks human-like scaling with capacity and task complexity.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Encoding mystery-box scenes as small fables with a good/bad question would let Mova's readers be scored against published human acceptance rates rather than against gold labels alone.
  • I2 (frames): The prag crate could be tested with minimal worlds where each implicature (ignorance, free choice, distributive) flips truth, yielding a precise check of which inference rule fired.
  • I3 (frames): Implicatures for "or" could be modelled as separate, individually explainable inferences rather than a single chained derivation, matching the human finding that POS and DI can appear alone.
  • I4 (learn): Evaluation could always report true/false control conditions next to each target test, acting as negative controls that expose systems accepting everything (as Mistral-Instruct did).
  • I5 (learn): When ranking LLM teacher tiers, Mova could weight a teacher's pragmatic labels by its agreement with human experimental data, not by model reputation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations🎓University of TübingenTübingen · Germany75 works · 41,081 citations🎓Heinrich Heine University DüsseldorfDüsseldorf · Germany33 works · 1,487 citations🎓University of MaltaMsida · Malta29 works · 894 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPhilosophy · 284 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksImplicature · 138 works

Cite

Cite as Polina Tsvilodub, Paul Marty, Sonia Ramotowska, Jacopo Romoli and Michael Franke. 2024. Experimental Pragmatics with Machines: Testing LLM Predictions for the Inferences of Plain and Embedded Disjunctions. arXiv:2405.05776.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.05776 · arXiv 2405.05776

@misc{tsvilodub2024experimental,
  title = {{Experimental Pragmatics with Machines: Testing LLM Predictions for the Inferences of Plain and Embedded Disjunctions}},
  author = {Polina Tsvilodub, Paul Marty, Sonia Ramotowska, Jacopo Romoli and Michael Franke},
  year = {2024},
  doi = {10.48550/arxiv.2405.05776},
  eprint = {2405.05776},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.05776}
}

References

32 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., . . . Penedo, G. (2023). Falcon-40B: an open large language model with state-of-the-art performance. find
  2. Aloni, M. (2022). Logic and conversation: the case of free choice. Semantics and Pragmatics, 15, 5–EA. ↗
  3. Bar-Lev, M. E., & Fox, D. (2020). Free choice, simplification, and innocent inclusion. Natural Language Semantics, 28(3), 175–223. ↗
  4. Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., . . . others (2023). Pythia: A suite for analyzing large language models across training and scaling. In International conference on machine learning (pp. 2397–2430). find
  5. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., . . . Amodei, D. (2020). Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, & H. Lin (Eds.), Advances in neural information processing systems (Vol. 33, pp. 1877–1901). Curran Associates, Inc.
  6. Crnič, L., Chemla, E., & Fox, D. (2015). Scalar implicatures of embedded disjunction. Natural Language Semantics, 23(4), 271–305. doi: 10.1007/s11050-015-9116-x ↗
  7. Degano, M., Ramotowska, S., Marty, P., Aloni, M., Breheny, R., Romoli, J., & Sudo, Y. (2023). The ups and downs of ignorance. Retrieved from https://ling.auf.net/lingbuzz/007389 (under review) ↗
  8. Fox, D. (2007). Free choice and the theory of scalar implicatures. In U. Sauerland & P. Stateva (Eds.), Presupposition and implicature in compositional semantics (pp. 71– 120). London: Palgrave Macmillan UK. Retrieved from doi: https://doi.org/10.1057/9780230210752 4 10.1057/9780230210752 4 ↗
  9. Gauthier, J., Hu, J., Wilcox, E., Qian, P., & Levy, R. (2020, July). SyntaxGym: An online platform for targeted evaluation of language models. In Proceedings of the 58th annual meeting of the association for computational linguistics: System demonstrations (pp. 70–76). Online: Association for Computational Linguistics.
  10. Grice, P. H. (1975). Logic and conversation. In P. Cole & J. L. Morgan (Eds.), Syntax and semantics, vol. 3, speech acts (pp. 41–58). New York: Academic Press. ↗
  11. Holtzman, A., West, P., Shwartz, V., Choi, Y., & Zettlemoyer, L. (2021). Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 conference on empirical methods in natural language processing (pp. 7038–7051). Association for Computational Linguistics.
  12. Hu, J., & Frank, M. C. (2024). Auxiliary task demands mask the capabilities of smaller language models. arXiv preprint arXiv:2404.02418. ↗
  13. Hu, J., Levy, R., & Schuster, S. (2022, May). Predicting scalar diversity with context-driven uncertainty over alternatives. In E. Chersoni, N. Hollenstein, C. Jacobs, Y. Oseki, L. Prévot, & E. Santus (Eds.), Proceedings of the workshop on cognitive modeling and computational linguistics (pp. 68–74). Dublin, Ireland: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2022.cmcl-1.8 doi: 10.18653/v1/2022.cmcl-1.8 ↗
  14. Jeretic, P., Warstadt, A., Bhooshan, S., & Williams, A. (2020, July). Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In D. Jurafsky, J. Chai, N. Schluter, & J. Tetreault (Eds.), Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 8690–8705). Online: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2020.acl-main.768 doi: 10.18653/v1/2020.acl-main.768 ↗
  15. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., . . . others (2023). Mistral 7B. arXiv preprint arXiv:2310.06825. ↗
  16. Kamp, H. (1978). Semantics versus pragmatics. In F. Guenthner & S. J. Schmidt (Eds.), Formal semantics and pragmatics for natural languages (pp. 255–287). Dordrecht: Springer Netherlands. Retrieved from https://doi.org/10.1007/978-94-009-9775-2 9 doi: 10.1007/978-94-009-9775-2 9 ↗
  17. Li, E., Schuster, S., & Degen, J. (2021). Predicting scalar inferences from “or” to “not both” using neural sentence encoders. In Proceedings of the society for computation in linguistics 2021 (pp. 446–450). ↗
  18. Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., & Lee, Y. T. (2023). Textbooks are all you need II: phi-1.5 technical report. arXiv preprint arXiv:2309.05463. ↗
  19. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. ↗
  20. Marty, P., Ramotowska, S., Romoli, J., Sudo, Y., & Breheny, R. (2023). On the source of distributive inferences. Retrieved from https://lingbuzz.net/lingbuzz/007623? s=6Vr1I-H6Shz7nms3w& k=-kuDvMqnl4piFlbG (under review) ↗
  21. Marty, P., Romoli, J., Sudo, Y., & Breheny, R. (2023). What makes an inference robust? Journal of Semantics. Retrieved from https://lingbuzz.net/lingbuzz/006205? s=pEQA-HafzjamqK7zo& k=6O7AfYta3cP962OR Miles, J. (2005). R-squared, adjusted r-squared. Encyclopedia of statistics in behavioral science. ↗
  22. MistralAI. (2023). Mixtral of experts. a high quality sparse mixture-of-experts. Retrieved from https://mistral.ai/news/mixtral-of-experts/ ↗
  23. Noveck, I. (2018). Experimental Pragmatics: The Making of a Cognitive Science. Cambridge University Press. find
  24. Parrish, A., Schuster, S., Warstadt, A., Agha, O., Lee, S.-H., Zhao, Z., . . . Linzen, T. (2021, November). NOPE: A corpus of naturally-occurring presuppositions in English. In A. Bisazza & O. Abend (Eds.), Proceedings of the 25th conference on computational natural language learning (pp. 349–366). Online: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2021.conll-1.28 doi: 10.18653/v1/2021.conll-1.28 ↗
  25. Sauerland, U. (2004). Scalar implicatures in complex sentences. Linguistics and Philosophy, 27, 367–391. ↗
  26. Schuster, S., Chen, Y., & Degen, J. (2020, July). Harnessing the linguistic signal to predict scalar inferences. In D. Jurafsky, J. Chai, N. Schluter, & J. Tetreault (Eds.), Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 5387–5403). Online: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2020.acl-main.479 doi: 10.18653/v1/2020.acl-main.479 ↗
  27. Sieker, J., & Zarrieß, S. (2023). When your language model cannot even do determiners right: Probing for antipresuppositions and the maximize presupposition! principle. In Proceedings of the 6th blackboxnlp workshop: Analyzing and interpreting neural networks for NLP (pp. 180– 198). find
  28. Sravanthi, S. L., Doshi, M., Kalyan, T. P., Murthy, R., Bhattacharyya, P., & Dabre, R. (2024). PUB: A pragmatics understanding benchmark for assessing llms’ pragmatics capabilities. arXiv preprint arXiv:2401.07078. ↗
  29. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., . . . others (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  30. Truong, T. H., Baldwin, T., Verspoor, K., & Cohn, T. (2023). Language models are not naysayers: An analysis of language models on negation benchmarks. arXiv preprint arXiv:2306.08189. ↗
  31. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., . . . Bowman, S. (2019). Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32.
  32. Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R. (2020). BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8, 377–392. Retrieved from https://aclanthology.org/2020.tacl-1.25 doi: 10.1162/tacl a 00321 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tsvilodub-2024-experimental-pragmatics-machines-testing-llm/