Elamur Research⌕
☰
home / papers / 2023 / Does GPT-4 surpass human performance in linguistic pragmatics?

Does GPT-4 surpass human performance in linguistic pragmatics?

Paper · 2023 · Ljubisa Bojic, Predrag Kovacevic and Milan Cabarkapa

Summary

The paper tests whether large language models can recover implied meaning in conversation, a skill central to linguistic pragmatics. The authors wrote short invented dialogues around Grice's Cooperative Principle and its four maxims, where a speaker flouts a maxim and the listener must infer an implicature. GPT-2, GPT-3, GPT-3.5, GPT-4 and Bard answered the same open questions as 147 people, who formed two groups: 71 participants, most of them students in Serbia, and 76 people from the U.S. whose first language is English. Linguists graded all responses and combined the grading with triangulation. The authors expected humans to win, but the data contradicted this. GPT-4 achieved the best score, placing above the strongest human, and the LLMs as a group averaged higher than either human cohort. The older GPT-2 and GPT-3, which lack instruction tuning, did markedly worse. The authors note limits: non-random sampling, unequal testing conditions, English-only items and scripted rather than natural dialogue. The result matters because it shows pragmatic inference can emerge from general training without a task-specific objective.

Key points

  • The test design is grounded in Grice's maxims (Quantity, Quality, Relation, Manner) and in Relevance Theory by Sperber and Wilson. It covers both conversational implicatures (for example, irony at a rainy bus stop) and conventional ones triggered by words like "but".
  • The top scores were GPT-4 at 4.80 and the best human at 4.55, followed by GPT-3.5 at 4.10, Bard at 3.75, GPT-3 at 3.25 and GPT-2 at 1.05.
  • Group averages were 3.39 for the LLMs, 2.80 for the Serbian students and 2.34 for the U.S. participants. In a joint ranking of 155 subjects, GPT-4 came first and the best human second.
  • Hypothesis H1, that humans would beat LLMs, was not supported.
  • The models had observed weaknesses:
    • They tended to give overly long answers.
    • They sometimes misjudged which speaker was responsible for the communicative problem.
    • Non-instruction-tuned models struggled with the prompt format.

Interesting ideas

  • I1. Build each test dialogue so that one named Gricean maxim is deliberately flouted, which gives every item a known "reason" for the implied meaning and makes the answer checkable.
  • I2. Grade not only the recovered meaning but also which participant caused the conversational difficulty. Attribution errors turned out to be a distinct failure mode, separate from misreading the message.
  • I3. Treat conventional implicature as lexically triggered and context-independent: "but" always signals contrast, unlike context-driven conversational implicature.
  • I4. Place models and humans from different populations on a single ranked leaderboard instead of reporting only averages. This exposes overlap between groups and shows where the best individual sits.
  • I5. Begin with controlled, invented dialogues to isolate variables, then plan validation on naturally occurring corpus discourse for ecological validity.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): crate prag could output the flouted maxim as an explicit, explainable label next to each inferred implicature, and these scripted dialogues could serve as unit tests for that label.
  • I2 (frames): implicature frames could carry a "source speaker" slot linked through coreference, and attribution accuracy could be scored separately from meaning accuracy.
  • I3 (frames): a small lexicon of trigger words could fire contrast or expectation frames directly from the UD tree before any contextual reasoning runs.
  • I4 (learn): Mova's readers, LLM teachers and human baselines could be ranked together on pragmatics items, revealing which teacher tier is actually reliable.
  • I5 (learn): synthetic maxim-flouting items could serve as clean regression and negative-control tests, while natural fable and dialogue text checks that gains transfer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🤝Complexity Science HubVienna · Austria3 works · 279 citations🎓University of BelgradeBelgrade · Serbia2 works · 23 citations

Also on the byline: University of Kragujevac · University of Novi Sad · Institute for Philosophy and Social Theory · Istraživačko-razvojni institut za veštačku inteligenciju Srbije.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Ljubisa Bojic, Predrag Kovacevic and Milan Cabarkapa. 2023. Does GPT-4 surpass human performance in linguistic pragmatics?. arXiv:2312.09545.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.09545 · arXiv 2312.09545

@misc{bojic2023does,
  title = {{Does GPT-4 surpass human performance in linguistic pragmatics?}},
  author = {Ljubisa Bojic, Predrag Kovacevic and Milan Cabarkapa},
  year = {2023},
  doi = {10.48550/arxiv.2312.09545},
  eprint = {2312.09545},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.09545}
}

References

48 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdulhai, M., Levine, S. & Jaques, N. (2022). Moral Foundations of Large Language Models.
  2. The AAAI 2023 Workshop on Representation Learning for Responsible Human-Centric AI (R2HCAI). https://r2hcai.github.io/AAAI-23/files/CameraReadys/49.pdf ↗
  3. Alipour, S., Galeazzi, A., Sangiorgio, E., Avalle, M., Bojic, L., Cinelli, M., & Quattrociocchi, W. (2024). Cross-platform social dynamics: An analysis of ChatGPT and COVID-19 vaccine conversations. Scientific Reports, 14(1), 2789. https://doi.org/10.1038/s41598-024-53124-x ↗
  4. Amirizaniani, M., Martin, E., Sivachenko, M., Mashhadi, A., & Shah, C. (2024). Can LLMs reason like humans? Assessing theory of mind reasoning in llms for open-ended questions. find
  5. Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, 34–44. https://doi.org/10.1145/3627673.3679832 Austin, J. L. (1962). How To Do Things With Words. Oxford University Press. ↗
  6. Barnard, M. & Otte, W. (2024). Is the Machine Surpassing Humans?: Large Language Models, Structuralism, and Liturgical Ritual: A Position Paper. International Journal of Practical Theology, 28(2), 289-306. https://doi.org/10.1515/ijpt-2023-0078 ↗
  7. Binz, M., & Schulz, E. (2023). Using cognitive psychology to understand GPT-3. ↗
  8. Proceedings of the National Academy of Sciences, 120(6), e2218523120. https://doi.org/10.1073/pnas.2218523120 Bojic, L. (2022). Metaverse through the prism of power and addiction: What will happen when the virtual world becomes more attractive than reality? European Journal of Futures Research, 10(1), 22. https://doi.org/10.1186/s40309-022-00208-4 ↗
  9. Bojic, L. (2024). AI alignment: Assessing the global impact of recommender systems. ↗
  10. Futures, 160, 103383. https://doi.org/10.1016/j.futures.2024.103383 ↗
  11. Bojic, L., Cinelli, M., Culibrk, D. Delibasic, B. (2024a). CERN for AI: a theoretical framework for autonomous simulation-based artificial intelligence testing and alignment. ↗
  12. European Journal of Futures Research, 12, 15. https://doi.org/10.1186/s40309-024-00238-0 Bojić, L., Stojković, I. & Jolić Marjanović, Z. (2024b). Signs of consciousness in AI: Can GPT-3 tell how smart it really is?. Humanities and Social Sciences Communications, 11, 1631. https://doi.org/10.1057/s41599-024-04154-3 ↗
  13. Bojić, L., Agatonović, M., Guga, J. (2024c). The Immersion in the Metaverse: Cognitive Load and Addiction. In: Geroimenko, V. (eds) Augmented and Virtual Reality in the Metaverse. ↗
  14. Springer Series on Cultural Computing. Springer, Cham. https://doi.org/10.1007/978-3-031-57746-8_11 Brown, P., & Levinson, S. (1987). Politeness: Some Universals in Language Use. ↗
  15. Cambridge University Press. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners (arXiv:2005.14165). arXiv. http://arxiv.org/abs/2005.14165 ↗
  16. Bodroza, B., Dinic, B. M., & Bojic, L. (2023). Personality testing of GPT-3: Limited temporal reliability, but highlighted social desirability of GPT-3’s personality instruments results. arXiv.Org. https://arxiv.org/abs/2306.04308v1 ↗
  17. Gao, C., Lan, X., Li, N., Yuan, Y., Ding, J., Zhou, Z., Xu, F., & Li, Y. (2024). Large language models empowered agent-based modeling and simulation: a survey and perspectives. ↗
  18. Humanities and Social Sciences Communications, 11, 1259. https://doi.org/10.1057/s41599-024-03611-3 Grice, H. P. (1975). Logic and Conversation. In P. Cole & J. L. Morgan (Eds.), Syntax and Semantics (Vol. 3). Academic Press. ↗
  19. Grice, H. P. (1989). Studies in the Way of Words. Harvard University Press. find
  20. Hartmann, J., Schwenzow, J., & Witte, M. (2023). The political ideology of conversational AI: Converging evidence on ChatGPT’s pro-environmental, left-libertarian orientation (arXiv:2301.01768). arXiv. http://arxiv.org/abs/2301.01768 ↗
  21. Hymes, D. (1972), “On communicative competence”, in Pride, J.B. and Holmes, J. (Eds), Sociolinguistics, Penguin, Harmondsworth, pp. 269-293. find
  22. Huang, Y. (2006). Pragmatics. Oxford University Press. find
  23. King, M. (2023). GPT-4 aligns with the New Liberal Party, while other large language models refuse to answer political questions. Engineering Archive. https://doi.org/10.31224/2974 ↗
  24. Kosinski, M. (2023). Theory of Mind May Have Spontaneously Emerged in Large Language Models. ArXiv. https://doi.org/10.48550/arXiv.2302.02083 ↗
  25. Kumar, H., Musabirov, I., Shi, J., Lauzon, A., Choy, K.K., Gross, O., Kulzhabayeva, D., & Williams, J.J. (2022). Exploring the Design of Prompts for Applying GPT-3 based Chatbots: A Mental Wellbeing Case Study on Mechanical Turk. ArXiv. https://doi.org/10.48550/arXiv.2209.11344 ↗
  26. Latenta (2025, January 29). Report for Interpretations Survey 2025. https://reporting.alchemer.eu/r/90032655_679900c395dc16.68572906 ↗
  27. Levinson, S. C. (1983). Pragmatics. Cambridge University Press. find
  28. Li, X., Li, Y., Liu, L., Bing, L. & Joty, S. (2022). Is GPT-3 a Psychopath? Evaluating Large Language Models from a Psychological Perspective. ArXiv. https://doi.org/10.48550/arXiv.2212.10529 ↗
  29. McCarthy, J., & Hayes, P. J. (1981). Some Philosophical Problems from the Standpoint of Artificial Intelligence. Machine Intelligence, 4(463-502). ↗
  30. McGee, R. W. (2023). Capitalism, socialism and chatgpt (SSRN Scholarly Paper 4369953). https://doi.org/10.2139/ssrn.4369953 ↗
  31. Minsky, M. (1981). A Framework for Representing Knowledge. In: Haugeland, J. (ed) find
  32. Mind Design. The MIT Press. OSF (2023). Linguistic pragmatics, the Generative Pre-training Transformer and Bard families. https://osf.io/96wrq/?view_only=21e230071e824c01be6c075e98d79503 ↗
  33. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf ↗
  34. Russell, S. J., & Norvig, P. (2010). Artificial Intelligence: A Modern Approach (3rd ed.). find
  35. Prentice Hall. Rutinowski, J., Franke, S., Endendyk, J., Dormuth, I., & Pauly, M. (2023). The self-perception and political biases of chatgpt (arXiv:2304.07333). arXiv. http://arxiv.org/abs/2304.07333 ↗
  36. Searle, J. R. (1969). Speech Acts: An Essay in the Philosophy of Language (Vol. 626). ↗
  37. Cambridge University Press. Simmons, G. (2023). Moral mimicry: Large language models produce moral rationalizations tailored to political identity (arXiv:2209.12106). arXiv. http://arxiv.org/abs/2209.12106 ↗
  38. Skjuve, M., Følstad, A., Fostervold, K.I., & Brandtzaeg, P.B. (2022). A longitudinal study of human–chatbot relationships. International Journal of Human-Computer Studies, 168, 102903. doi: https://doi.org/10.1016/j.ijhcs.2022.102903 ↗
  39. Sperber, D., & Wilson, D. (1986). Relevance: Communication and Cognition (1st ed.). find
  40. Blackwell Publishing. Sperber, D., & Wilson, D. (1995). Relevance: Communication and Cognition. Blackwell Publishing. https://monoskop.org/images/e/e6/Sperber_Dan_Wilson_Deirdre_Relevance_Communica_and_ ↗
  41. Cognition_2nd_edition_1996.pdf Stefanowicz, B. (2022, October 11). 23 Top Real-Life Chatbot Use Cases That Work. find
  42. Tidio. https://www.tidio.com/blog/chatbot-use-cases/Thomas, J. (1995). Meaning in Interaction: An Introduction to Pragmatics. Longman. ↗
  43. Thunström, A. O. (2022, June 30th). We Asked GPT-3 to Write an Academic Paper about Itself—Then We Tried to Get It Published. Scientific American. https://www.scientificamerican.com/article/we-asked-gpt-3-to-write-an-academic-paper-about-itself-mdash-then-we-tried-to-get-it-published/ ↗
  44. Wu, P. Y., Nagler, J., Tucker, J. A., & Messing, S. (2023). Large language models can be used to estimate the latent positions of politicians (arXiv:2303.12057). arXiv. http://arxiv.org/abs/2303.12057 ↗
  45. Xie, C., & Zou, D. (2024). A human-like reasoning framework for multi-phases planning task with large language models (arXiv:2405.18208). arXiv. https://doi.org/10.48550/arXiv.2405.18208 ↗
  46. Yule, G. (2006). Pragmatics. Oxford University Press. find
  47. Zhang, M., & Li, J. (2021). A commentary of GPT-3 in MIT Technology Review 2021. ↗
  48. Fundamental Research, 1(6), 831–833. https://doi.org/10.1016/j.fmre.2021.11.011 Ethical Approval ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bojic-2023-gpt-4-surpass-human-performance/