Elamur Research⌕
☰
home / papers / 2024 / Pragmatic inference of scalar implicature by LLMs

Pragmatic inference of scalar implicature by LLMs

Summary

This student workshop paper asks whether early transformer LLMs read the quantifier some literally ("at least one, maybe all") or pragmatically ("not all"). It also asks whether conversational context shifts that reading. The authors pulled sentences of the form 'some + NP' from the British National Corpus (BNC) with NLTK. They then ran two experiments on BERT and GPT-2, modeled on studies of human processing.

  • Experiment 1 (no context): cosine similarity between each some-sentence and a semantic paraphrase ("Possibly all…") versus a pragmatic one ("Not all…"). Both models preferred the pragmatic reading, as humans do.
  • Experiment 2 (context): the sentences were preceded by a Question Under Discussion (QUD), either upper-bound (all) or lower-bound (any), and scored with next sentence/token prediction. BERT behaved the same under both QUD types, which fits the Default model (Levinson, 2000). GPT-2 struggled with the QUD type that demands implicature, which matches the Context-driven model (Sperber and Wilson, 2002).

The work brings context-sensitive pragmatics into LLM evaluation.

Key points

  • Data: subject-position 'some + NP' sentences from the BNC, extracted with NLTK. The data and results are released publicly.
  • Methods: cosine similarity (Experiment 1) and next sentence/token prediction (Experiment 2) on BERT and GPT-2.
  • Without context, both models favour the not all implicature over the semantic reading.
  • Under QUD context, BERT is insensitive to the question type (Default model). GPT-2 has difficulty when an upper-bound QUD demands the implicature (Context-driven model).
  • Limitations noted by the authors:
    • Only one construction is tested: single-clause sentences with some in subject position.
    • The dataset is modest in size.
    • Only two older models are used.
    • The metrics are non-standard compared with NLI setups.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Implicature

Cited by

13 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 11 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-srw.2 · arXiv 2408.06673

@inproceedings{cho2024pragmatic,
  title = {{Pragmatic inference of scalar implicature by LLMs}},
  author = {Cho, Ye-eun and Kim, Seong mook},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-srw.2},
  eprint = {2408.06673},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.acl-srw.2}
}

References

34 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ludivine E. Dupuy, Jean-Baptiste Van der Henst, Anne Cheylus and Anne C. Reboul. 2016. Context in generalized conversational implicatures: the case of some. Frontiers in Psychology, 7:381. ↗
  2. Douglas Bates, Martin Mächler, Benjamin M. Bolker and Steven C. Walker. 2014. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67:1-48. ↗ our card
  3. Bart Geurts. 2010. Quantity implicatures. Cambridge University Press. find
  4. David Beaver and Brady Clark. 2008. Sense and sensitivity: How focus determines meaning. John Wiley & Sons. find
  5. Bart Geurts, Napoleon Katsos, Chris Cummins, Jonas Moons and Leo Noordman. 2010. Scalar quantifiers: Logic, acquisition, and processing. Language and cognitive processes, 25(1):130-148. ↗
  6. Steven Bird. 2006. NLTK: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions. pages 69-72. 18 9 find
  7. Bart Geurts and Rick Nouwen. 2007. 'At least'et al.: the semantics of scalar modifiers. Language, 83(3):533-559. find
  8. Stephen Politzer-Ahles and Robert Fiorentino. 2013. The realization of scalar inferences: Context sensitivity without processing cost. PloS ONE, 8(5):e63943. ↗
  9. Herbert P Grice. 1975. Logic and conversation. In Speech acts (pp. 41-58). Brill. ↗
  10. Stephen Politzer-Ahles and E. Matthew Husband. 2018. Eye movement evidence for context-sensitive derivation of scalar inferences. Collabra: Psychology, 4(1):3. ↗
  11. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second meeting of the north american chapter of the association for computational linguistics. (Volume 2). pages 159-166. find our card
  12. Jeffrey Pennington, Richard Socher and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pages 1532-1543. find
  13. Laurence Robert Horn. 1972. On the semantic properties of logical operators in English. University of California, Los Angeles. find
  14. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  15. Jennifer Hu, Roger Levy, Judith Degen and Sebastian Schuster. 2023. Expectations over unspoken alternatives predict pragmatic inferences. Transactions of the Association for Computational Linguistics, 11, 885-901. ↗ our card
  16. Craige Roberts. 2004. Discourse context in dynamic interpretation. In L. Horn & G. Ward (Eds.), Handbook of contemporary pragmatic theory (pp. 197–220). Oxford: Blackwell. find
  17. Yi Ting Huang and Jesse Snedeker. 2009. Online interpretation of scalar quantifiers: Insight into the semantics–pragmatics interface. Cognitive psychology, 58(3):376-415. ↗
  18. Craige Roberts. 2012. Information structure: Towards an integrated formal theory of pragmatics. Semantics and Pragmatics, 5(6):1-69. find
  19. Lamar Hunt III, Stephen Politzer-Ahles, Linzi Gibson, Utako Minai and Robert Fiorentino. 2013. Pragmatic inferences modulate N400 during sentence comprehension: Evidence from picture– sentence verification. Neuroscience Letters, 534:246-251. ↗
  20. Eszter Ronai and Ming Xiang. 2021. Pragmatic inferences are QUD-sensitive: an experimental study. Journal of Linguistics, 57(4):841-870. ↗
  21. Sebastian Schuster, Yuxing Chen and Judith Degen. 2020. Harnessing the linguistic signal to predict scalar inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 5387-5403. find our card
  22. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan and Adina Williams. 2020. Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 8690-8705. find our card
  23. Dan Sperber and Deirdre Wilson. 2002. Pragmatics, modularity and mind-reading. Mind & Language, 17:3-23. ↗
  24. Stephen C Levinson. 2000. Presumptive meanings: The theory of generalized conversational implicature. MIT press. ↗
  25. John M. Tomlinson Jr, Todd M. Bailey and Lewis Bott. 2013. Possibly all of that and then some: Scalar implicatures are understood in two steps. Journal of memory and language, 69(1):18-35. ↗
  26. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126-1177. ↗
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. find
  28. Elissa Li, Sebastian Schuster and Judith Degen. 2021. Predicting scalar inferences from “or” to “not both” using neural sentence encoders. In Proceedings of the Society for Computation in Linguistics 2021. pages 446-450. ↗ our card
  29. Deirdre Wilson and Dan Sperber. 1995. Relevance theory. In L. Horn & G. Ward (Eds.) The handbook of pragmatics (pp. 606-632). Oxford: Blackwell. find
  30. Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv:1301.3781. ↗
  31. Ira Xiao Yang, Utako Minai and Robert Fiorentino. 2018. Context-sensitivity and individual differences in the derivation of scalar implicature. Frontiers in psychology, 9:1720. ↗
  32. A. Noveck and Andres Posada. 2003. Characterizing the time course of an implicature: An evoked potentials study. Brain and language, 85(2):203-210. 19 10 ↗
  33. Kaitlyn Zhou, Kawin Ethayarajh, Dallas Card and Dan Jurafsky. 2022. Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2). pages 401-423. find
  34. Arjen Zondervan, Luisa Meroni and Andrea Gualmini. 2008. Experiments on the role of the question under discussion for ambiguity resolution and implicature computation in adults. In Semantics and linguistic theory (Volume 18). pages 765-777. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cho-2024-pragmatic-inference-scalar-implicature-llms/