Elamur Research⌕
☰
home / papers / 2023 / Does ChatGPT Resemble Humans in Processing Implicatures?

Does ChatGPT Resemble Humans in Processing Implicatures?

Paper · 2023 · Zhuang Qiu, Xufeng Duan and Zhenguang Cai

Summary

The paper asks whether ChatGPT handles conversational implicatures the way people do. Implicatures are meanings a listener infers beyond the literal content, for example reading "most" as "not all". The authors ran three preregistered experiments on the Feb 13 version of ChatGPT and reused paradigms from human studies so the results could be compared directly.

  • Experiment 1 follows Doran et al. (2012). The model judged short dialogues against a stated fact under two instructions: a plain judgement, or a judgement of literal truth only. This tested whether it can switch off generalized conversational implicatures (GCIs) on request.
  • Experiments 2 and 3 asked whether communicative context changes how often scalar implicatures (SIs) are drawn.

Humans compute fewer GCIs when told to read literally, and they draw fewer SIs when the enriched reading would threaten the listener's face. ChatGPT did neither. It kept computing implicatures under literal instructions. Its scalar judgements were essentially deterministic: "some" was always read as "not all", and "or" was almost always read inclusively.

This matters because ChatGPT matches humans on many other psycholinguistic effects, yet here it shows a clear gap in context-sensitive pragmatic flexibility.

Key points

  • Experiment 1 used 44 critical items: three GCI families (Q-based, I-based, M-based) split into subcategories with four items each. It also included 16 fillers from Doran et al. (2012) that need no implicature.
  • Experiment 1 had 400 scripted runs, 200 in the pragmatic condition and 200 in the literal condition. Each run began with a training example, and the model was told to answer only "true" or "false".
  • The procedure was preregistered on the Open Science Framework (osf.io/cp29j).
  • In the face-management experiment (after Bonnefon, Feeney & Villejoubert, 2009), humans rejected the lower-bounded reading of "some" at different rates:
    • 83% in face-boosting contexts;
    • 58% in face-threatening contexts.
    • ChatGPT rejected it on every trial in both contexts, with zero variance.
  • The authors offer two explanations:
    • Humans learn literal scalar meanings before pragmatic enrichment, whereas LLM training text is mostly pragmatically loaded.
    • Human cognitive economy enriches meaning only when the context calls for it.

Interesting ideas

  • I1. Present the same item once with a neutral instruction and once with a "literally true?" instruction, then measure whether the system's verdicts shift; this tests controllability of inference, not just accuracy.
  • I2. Pair a speaker's utterance with an explicit FACT and ask whether the utterance is true; the gap between "logically true" and "pragmatically misleading" becomes directly measurable.
  • I3. Treat zero variance across a manipulated context as a red flag: if outputs never change when the condition changes, the system is ignoring that factor.
  • I4. Model scalar items as a basic literal (lower-bounded) meaning plus an enrichment that fires only when context supports it, rather than hard-wiring the "not all" reading.
  • I5. Horn-style scales such as some < most < all drive Q-based implicatures by negating the stronger alternatives the speaker could have used.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): crate prag could keep the literal truth value and each implicature as separate, labelled outputs, so a query can ask for either one and a regression test can confirm that the two actually diverge.
  • I2 (answer): the readers could be tested on items like this, checking that they report "true but underinformative" rather than collapsing it into a plain yes or no.
  • I3 (learn): evaluation could include paired context manipulations with known human direction effects, and flag any component whose output distribution stays the same across them.
  • I4 (frames): implicature rules in prag could start from the lower-bounded sense and attach the upper-bound inference with a confidence that politeness or face-threat cues can reduce.
  • I5 (world): a compiled table of lexical scales (quantifiers, "or/and", degree words) would let the world model generate and track these negated alternatives explicitly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Implicature · 138 works

Cite

Cite as Zhuang Qiu, Xufeng Duan and Zhenguang Cai. 2023. Does ChatGPT Resemble Humans in Processing Implicatures?. Proceedings of the 4th Natural Logic Meets Machine Learning Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{qiu2023does,
  title = {{Does ChatGPT Resemble Humans in Processing Implicatures?}},
  author = {Zhuang Qiu, Xufeng Duan and Zhenguang Cai},
  booktitle = {Proceedings of the 4th Natural Logic Meets Machine Learning Workshop},
  year = {2023},
  url = {https://aclanthology.org/2023.naloma-1.3/}
}

References

7 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. conversation. In Pragmatics, pp. 113-127. Brill 790 Laurence Robert Horn. 1972. On the semantic properties of logical operators in English. Gerry Altmann, and Mark Steedman.1988. Interaction 791 University of California, Los Angeles. 740 with context during human sentence processing. 792 741 Cognition 30, no. 3: 191-238. 793 Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Marcel Binz and Eric Schulz. 2023. Using cognitive 794 743 psychology to understand GPT-3. Proceedings of 795 744 the National Academy of Sciences 120, no. 6: 796 745 e2218523120. find
  2. Wang, and Zhaopeng Tu. 2023. Is ChatGPT a good translator? A preliminary study. arXiv preprint arXiv:2301.08745. 797 Michal Kosinski. 2023. Theory of mind may have spontaneously emerged in large language models. ↗
  3. J. Kathryn Bock. 1986. Syntactic persistence in 798 arXiv preprint arXiv:2302.02083. 747 language production. Cognitive psychology 18, no. 799 748 3: 355-387. 800 Stephen C Levinson. 2000. Presumptive meanings: The theory of generalized conversational Jean-François Bonnefon, Aidan Feeney, and Gaëlle 801 implicature. MIT press. ↗
  4. Villejoubert. 2009. When some is actually all: 802 751 Scalar inferences in face-threatening contexts. 803 Kyle Mahowald, Evelina Fedorenko, Steven T. 752 Cognition 112, no. 2: 249-258. 804 Piantadosi, and Edward Gibson. 2013. Info/information theory: Speakers choose shorter 753 Tom Brown, Benjamin Mann, Nick Ryder, Melanie 805 words in predictive contexts. Cognition 126, no. 2 754 Subbiah, Jared D. Kaplan, Prafulla Dhariwal, 806 (2013): 313-318. 755 Arvind Neelakantan et al. 2020. Language models 807 756 are few-shot learners. Advances in neural 808 Ira A Noveck. 2001. When children are more logical 757 information processing systems 33: 1877-1901. 809 than adults: Experimental investigations of scalar 810 implicature. Cognition 78, no. 2: 165-188. 758 Eric Brunet-Gouet, Nathan Vidal, and Paul Roux. 759 2023. Do conversational agents have a theory of 811 Ira A Noveck and Dan Sperber. 2007. The why and how 760 mind? A single case study of ChatGPT with the 812 of experimental pragmatics: the case of ‘scalar 761 Hinting, False Beliefs and False Photographs, and 813 inferences, in Advances in Pragmatics, ed N. 762 Strange Stories paradigms. HAL Open Science. 814 Burton-Roberts (Basingstoke: Palgrave), 184–212. 763 https://hal.science/hal-03991530/ 815 Miguel Ortega-Martín, Óscar García-Sierra, Alfonso 764 Zhenguang G Cai, David A. Haslett, Xufeng Duan, 816 Ardoiz, Jorge Álvarez, Juan Carlos Armenteros, and 765 Shuqi Wang, and Martin J. Pickering. 2023. Does 817 Adrián Alonso. 2023. Linguistic ambiguity analysis 766 ChatGPT resemble humans in language use? arXiv 818 in ChatGPT. arXiv preprint arXiv:2302.06426 767 preprint arXiv:2303.08014. 819 Steven T Piantadosia. 2023. Modern language models 768 Kimberly Wright Cassidy, Michael H. Kelly, and Lee'at 820 refute Chomsky’s approach to language. Lingbuzz 769 J. Sharoni. 1999. Inferring gender from name 821 Preprint, lingbuzz/007180 ↗
  5. Jennifer M. Rodd, Belen Lopez Cutrin, Hannah Kirsch, Alessandra Millar, and Matthew H. Davis. 2013. 824 Long-term priming of the meanings of ambiguous 825 words. Journal of Memory and Language 68, no. 2 826 (2013): 180-198. find
  6. Chris Westbury. 2005. Implicit sound symbolism in lexical access: Evidence from an interference task. 829 Brain and language 93, no. 1 (2005): 10-19. ↗
  7. Arjen Zondervan. 2010. Scalar implicatures or focus: an experimental approach. Netherlands Graduate 832 School of Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qiu-2023-does-chatgpt-resemble-humans-processing/