Elamur Research⌕
☰
home / papers / 2023 / Do large language models resemble humans in language use?

Do large language models resemble humans in language use?

Paper · 2023 · Zhenguang G. Cai, Xufeng Duan, David A. Haslett et al.

Summary

The paper asks whether chatbots built on large language models show the behavioural regularities that psycholinguists have documented in people. This bears on the debate over whether such systems can serve as models of human language cognition. The authors adapted 12 classic experimental paradigms, two for each of six levels: sounds, words, syntax, meaning, discourse and interlocutor sensitivity. They preregistered all of them and presented every item 1,000 times to ChatGPT and to the open-source Vicuna (13B). Responses were coded and analysed with mixed-effects regression, and in some cases compared directly against existing human datasets. ChatGPT matched the human pattern in 10 of the 12 studies and Vicuna in 7. Both models failed to favour shorter words in predictable contexts, and both ignored discourse context when resolving attachment ambiguity. The results suggest that many processing tendencies can arise from statistical learning on text alone. The battery also offers a reusable template for testing language models, including those for other languages, against human behaviour.

Key points

  • Shared successes (both models):
    • sound–shape matching (e.g., maluma vs takete); ChatGPT gave round responses at 0.79 vs 0.49, Vicuna at 0.38 vs 0.32
    • linking novel names to gender
    • word-meaning priming
    • structural priming of PO/DO datives, which was stronger when prime and target shared the verb
    • implicit causality driven by verb type
    • choosing American vs British meanings and words depending on the interlocutor
  • ChatGPT only:
    • noisy-channel reinterpretation of implausible DO sentences
    • susceptibility to semantic illusions (e.g., the Snoopy "cat" question)
    • distinguishing bridging from elaborative inferences, at 0.49 vs 0.23 "yes" responses
  • Failures in both models:
    • no shortening of words such as math/mathematics when the context is predictive
    • no use of the number of referents to resolve PP attachment
  • Dialect sensitivity: Both LLMs reacted more strongly than humans to the speaker's dialect. In ChatGPT this effect weakened across trials. The authors caution that the models were told the dialect explicitly, whereas humans inferred it from accent.
  • Materials: ChatGPT (9 Jan 2023 version) was tested. Preregistrations, data and code are on OSF: osf.io/vu2h3 (ChatGPT) and osf.io/sygku (Vicuna).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

 Fields, topics and concepts

Cite

Cite as Zhenguang G. Cai, Xufeng Duan, David A. Haslett, Shuqi Wang and Martin J. Pickering. 2023. Do large language models resemble humans in language use?. arXiv:2303.08014.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.08014 · arXiv 2303.08014

@misc{cai2023do,
  title = {{Do large language models resemble humans in language use?}},
  author = {Zhenguang G. Cai, Xufeng Duan, David A. Haslett, Shuqi Wang and Martin J. Pickering},
  year = {2023},
  doi = {10.48550/arxiv.2303.08014},
  eprint = {2303.08014},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.08014}
}

References

70 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Altmann, G., & Steedman, M. (1988). Interaction with context during human sentence processing. Cognition, 30(3), 191–238. https://doi.org/10.1016/0010-0277(88)90020-0 — [doi:10.1016/0010-0277(88)90020-0](https://doi.org/10.1016/0010-0277%2888%2990020-0) ↗
  2. Aher, G. V., Arriaga, R. I., & Kalai, A. T. (2023, July). Using large language models to simulate multiple humans and replicate human subject studies. Proceedings of the 40th International Conference on Machine Learning, PMLR 202:337-371. https://proceedings.mlr.press/v202/aher23a.html ↗
  3. Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337-351. https://doi.org/10.1017/pan.2023.2 ↗
  4. Bender, E. M., & Koller, A. (2020). Climbing towards NLU: On meaning, form, and understanding in the age of data. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5185–5198. https://aclanthology.org/2020.acl-main.463 ↗
  5. Bock, J. K. (1986). Syntactic persistence in language production. Cognitive Psychology, 18(3), 355–387. https://doi.org/10.1016/0010-0285(86)90004-6 — [doi:10.1016/0010-0285(86)90004-6](https://doi.org/10.1016/0010-0285%2886%2990004-6) ↗
  6. Brown, R., & Fish, D. (1983). The psychological causality implicit in language. Cognition, 14(3), 237–273. https://doi.org/10.1016/0010-0277(83)90006-9 Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-V… — [doi:10.1016/0010-0277(83)90006-9](https://doi.org/10.1016/0010-0277%2883%2990006-9) ↗
  7. Cai, Z. G. (2022). Interlocutor modelling in comprehending speech from interleaved interlocutors of different dialectic backgrounds. Psychonomic Bulletin & Review, 29(3), 1026–1034. https://doi.org/10.3758/s13423-022-02055-7 Cai, Z. G., Dunn, M. S., & Branigan, H. P. (accepted in principle). How do… — [doi:10.3758/s13423-022-02055-7](https://doi.org/10.3758/s13423-022-02055-7) ↗
  8. Cai, Z. G., Gilbert, R. A., Davis, M. H., Gaskell, M. G., Farrar, L., Adler, S., & Rodd, J. M. (2017). Accent modulates access to word meaning: Evidence for a speaker-model account of spoken word recognition. Cognitive Psychology, 98, 73–101. https://doi.org/10.1016/j.cogpsych.2017.08.003 — [doi:10.1016/j.cogpsych.2017.08.003](https://doi.org/10.1016/j.cogpsych.2017.08.003) ↗
  9. Cai, Z. G., & Zhao, N. (2019). The sound of gender: Inferring the gender of names in a foreign language. Journal of Cultural Cognitive Science, 3, 63–73. https://doi.org/10.1007/s41809-019-00028-2 — [doi:10.1007/s41809-019-00028-2](https://doi.org/10.1007/s41809-019-00028-2) ↗
  10. Cai, Z. G., Zhao, N., & Pickering, M. J. (2022). How do people interpret implausible sentences? Cognition, 225, 105101. https://doi.org/10.1016/j.cognition.2022.105101 — [doi:10.1016/j.cognition.2022.105101](https://doi.org/10.1016/j.cognition.2022.105101) ↗
  11. Cai, Z. G., & Zhao, N. (2024). Structural priming: An experimental paradigm for mapping linguistic representations. Language and Linguistics Compass, 18(2), e12507. https://doi.org/10.1111/lnc3.12507 — [doi:10.1111/lnc3.12507](https://doi.org/10.1111/lnc3.12507) ↗
  12. Cassani, G., Chuang, Y.-Y., & Baayen, R. H. (2020). On the semantics of nonwords and their lexical category. Journal of Experimental Psychology: Learning, Memory, and Cognition, 46(4), 621. https://doi.org/10.1037/xlm0000747 — [doi:10.1037/xlm0000747](https://doi.org/10.1037/xlm0000747) ↗
  13. Cassidy, K. W., Kelly, M. H., & Sharoni, L. J. (1999). Inferring gender from name phonology. Journal of Experimental Psychology: General, 128(3), 362–381. https://doi.org/10.1037/0096-3445.128.3.362 — [doi:10.1037/0096-3445.128.3.362](https://doi.org/10.1037/0096-3445.128.3.362) ↗
  14. Caucheteux, C., & King, J.-R. (2022). Brains and algorithms partially converge in natural language processing. Communications Biology, 5(1), 134. https://doi.org/10.1038/s42003-022-03036-1 — [doi:10.1038/s42003-022-03036-1](https://doi.org/10.1038/s42003-022-03036-1) ↗
  15. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., & Gonzalez, J. E. (2023). Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https://lmsys.org/blog/2023-03-30-vicuna/ ↗
  16. Chomsky, N. (2000). New horizons in the study of language and mind. find
  17. Cambridge University Press. Chomsky, N., Roberts, I., & Watumull, J. (2023, March 8). Noam Chomsky: The false promise of ChatGPT. The New York Times. https://archive.is/AgWkn\#selection-317.0-317.13 ↗
  18. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30. https://proceedings.neurips.cc/paper_files/paper/2017/hash/d5e2c0adad503c91f91df240d 0cd4e49-Abstract.html Co… ↗
  19. Cutler, A., McQueen, J., & Robinson, K. (1990). Elizabeth and John: Sound patterns of men’s and women’s names. Journal of Linguistics, 26(2), 471–482. https://doi.org/doi:10.1017/S0022226700014754 — [doi:10.1017/s0022226700014754](https://doi.org/10.1017/s0022226700014754) ↗
  20. Ćwiek, A., Fuchs, S., Draxler, C., Asu, E. L., Dediu, D., Hiovain, K., Kawahara, S., Koutalidis, S., Krifka, M., & Lippus, P. (2022). The bouba/kiki effect is robust across cultures and writing systems. Philosophical Transactions of the Royal Society B, 377(1841), 20200390. https://doi.org/10.1098/r… — [doi:10.1098/rstb.2020.0390](https://doi.org/10.1098/rstb.2020.0390) find
  21. Dentella, V., Günther, F., & Leivada, E. (2023). Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response bias. Proceedings of the National Academy of Sciences, 120(51), e2309583120. find
  22. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv. https://doi.org/10.48550/arXiv.1810.04805 ↗
  23. Dunn, M. S., & Cai, Z. G. (2023). Word length affects language production in (non)predictive contexts but not language comprehension [poster presentation]. find
  24. AMLaP Asia, Hong Kong. Erickson, T. D., & Mattson, M. E. (1981). From words to meaning: A semantic illusion. Journal of Verbal Learning and Verbal Behavior, 20(5), 540–551. https://doi.org/10.1016/S0022-5371(81)90165-1 — [doi:10.1016/s0022-5371(81)90165-1](https://doi.org/10.1016/s0022-5371%2881%2990165-1) ↗
  25. Ethayarajh, K. (2019). How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. arXiv. https://doi.org/10.48550/arXiv.1909.00512 ↗
  26. Fukumura, K., & van Gompel, R. P. G. (2010). Choosing anaphoric expressions: Do people take into account likelihood of reference? Journal of Memory and Language, 62(1), 52–66. https://doi.org/10.1016/j.jml.2009.09.001 — [doi:10.1016/j.jml.2009.09.001](https://doi.org/10.1016/j.jml.2009.09.001) ↗
  27. Garvey, C., & Caramazza, A. (1974). Implicit causality in verbs. Linguistic Inquiry, 5(3), 459– 464. find
  28. Gatti, D., Marelli, M., & Rinaldi, L. (2022). Out-of-vocabulary but not meaningless: Evidence for semantic-priming effects in pseudoword processing. Journal of Experimental Psychology: General, 152(3), 851–863. https://doi.org/10.1037/xge0001304 — [doi:10.1037/xge0001304](https://doi.org/10.1037/xge0001304) ↗
  29. Gibson, E., Bergen, L., & Piantadosi, S. T. (2013). Rational integration of noisy evidence and prior semantic expectations in sentence interpretation. Proceedings of the National Academy of Sciences, 110(20), 8051–8056. https://doi.org/10.1073/pnas.1216438110 — [doi:10.1073/pnas.1216438110](https://doi.org/10.1073/pnas.1216438110) ↗
  30. Gilbert, R. A., & Rodd, J. M. (2022). Dominance norms and data for spoken ambiguous words in British English. Journal of Cognition, 5(1), 4. https://doi.org/10.5334/joc.194 — [doi:10.5334/joc.194](https://doi.org/10.5334/joc.194) ↗
  31. Goldberg, Y. (2019). Assessing BERT's syntactic abilities. https://doi.org/10.48550/arXiv.1901.05287 ↗
  32. Gulordava, K., Bojanowski, P., Grave, E., Linzen, T., & Baroni, M. (2018). Colorless green recurrent networks dream hierarchically. arXiv. https://arxiv.org/abs/1803.11138 ↗
  33. Hannon, B., & Daneman, M. (2001). Susceptibility to semantic illusions: An individual-differences perspective. Memory & Cognition, 29(3), 449–461. https://doi.org/10.3758/BF03196396 — [doi:10.3758/bf03196396](https://doi.org/10.3758/bf03196396) ↗
  34. Haslett, D. A., & Cai, Z. G. (2023). Similar-sounding words flesh out fuzzy meanings. Journal of Experimental Psychology: General. 152(8), 2359– 2368. https://doi.org/10.1037/xge0001409 — [doi:10.1037/xge0001409](https://doi.org/10.1037/xge0001409) ↗
  35. Hughes, A. (2023). ChatGPT: Everything you need to know about OpenAI’s GPT-4 tool. BBC Science Focus Magazine. https://www.sciencefocus.com/future-technology/gpt-3/ ↗
  36. Jain, S., Vo, V. A., Wehbe, L., & Huth, A. G. (2023). Computational language modeling and the promise of in silico experimentation. Neurobiology of Language, 1-27. https://doi.org/10.1162/nol_a_00101 Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., ... & Fung, P. (2023). Survey of hallucination… ↗
  37. Kaushal, A., & Mahowald, K. (2022). What do tokens know about their characters and how do they know it? arXiv. https://doi.org/10.48550/arXiv.2206.02608 ↗
  38. Köhler, W. (1929). Gestalt Psychology. find
  39. Liveright. Levy, R., Bicknell, K., Slattery, T., & Rayner, K. (2009). Eye movement evidence that readers maintain and act on uncertainty about past linguistic input. Proceedings of the National Academy of Sciences, 106(50), 21086–21090. https://doi.org/10.1073/pnas.0907664106 — [doi:10.1073/pnas.0907664106](https://doi.org/10.1073/pnas.0907664106) ↗
  40. Linzen, T., & Baroni, M. (2021). Syntactic Structure from Deep Learning. Annual Review of Linguistics, 7(1), 195–212. https://doi.org/10.1146/annurev-linguistics-032020-051035 ↗
  41. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv. https://doi.org/10.48550/arXiv.1907.11692 ↗
  42. Mahowald, K. (2023). A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Construction. arXiv. https://doi.org/10.48550/arXiv.2301.12564 ↗
  43. Mahowald, K., Fedorenko, E., Piantadosi, S. T., & Gibson, E. (2013). Info/information theory: Speakers choose shorter words in predictive contexts. Cognition, 126(2), 313–318. https://doi.org/10.1016/j.cognition.2012.09.010 — [doi:10.1016/j.cognition.2012.09.010](https://doi.org/10.1016/j.cognition.2012.09.010) ↗
  44. Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2023). Dissociating language and thought in large language models: A cognitive perspective. arXiv. http://arxiv.org/abs/2301.06627 ↗
  45. Marvin, R., & Linzen, T. (2018). Targeted syntactic evaluation of language models. arXiv. https://arxiv.org/abs/1808.09031 ↗
  46. McCoy, R. T., Pavlick, E., & Linzen, T. (2019). Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv. https://doi.org/10.48550/arXiv.1902.01007 ↗
  47. McKoon, G., & Ratcliff, R. (1986). Inferences about predictable events. Journal of Experimental Psychology: Learning, Memory, and Cognition, 12, 82–91. https://doi.org/10.1037/0278-7393.12.1.82 — [doi:10.1037/0278-7393.12.1.82](https://doi.org/10.1037/0278-7393.12.1.82) ↗
  48. Michaelov, J., Arnett, C., Chang, T., & Bergen, B. (2023). Structural priming demonstrates abstract grammatical representations in multilingual language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 3703–3720. https://aclanthology.org/2023.emnlp-main… ↗
  49. OpenAI. (2022, November 30). Introducing ChatGPT. https://openai.com/blog/chatgpt ↗
  50. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., & Ray, A. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35. https://proceedings.neurips.cc/paper_files/paper/2… ↗
  51. Piantadosi, S. (2023). Modern language models refute Chomsky’s approach to language. Lingbuzz Preprint, 7180. https://lingbuzz.net/lingbuzz/007180/v1.pdf ↗
  52. Piantadosi, S. T., Tily, H., & Gibson, E. (2011). Word lengths are optimized for efficient communication. Proceedings of the National Academy of Sciences, 108(9), 3526–3529. https://doi.org/10.1073/pnas.1012551108 — [doi:10.1073/pnas.1012551108](https://doi.org/10.1073/pnas.1012551108) ↗
  53. Pickering, M. J., & Branigan, H. P. (1998). The representation of verbs: Evidence from syntactic priming in language production. Journal of Memory and Language, 39(4), 633–651. https://doi.org/10.1006/jmla.1998.2592 — [doi:10.1006/jmla.1998.2592](https://doi.org/10.1006/jmla.1998.2592) ↗
  54. Prasad, G., Van Schijndel, M., & Linzen, T. (2019). Using priming to uncover the organization of syntactic representations in neural language models. arXiv. https://doi.org/10.48550/arXiv.1909.10579 ↗
  55. Qiu, Z., Duan, X., & Cai, Z. G. (2023). Pragmatic Implicature Processing in ChatGPT. PsyArXiv. https://doi.org/10.31234/osf.io/qtbh9. — [doi:10.31234/osf.io/qtbh9](https://doi.org/10.31234/osf.io/qtbh9) ↗
  56. Qiu, Z., Duan, X., & Cai, Z. G. (2024). Grammaticality Representation in ChatGPT as Compared to Linguists and Laypeople. Retrieved from osf.io/preprints/psyarxiv/r9zdh Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. http… find
  57. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf ↗
  58. Rayner, K., Carlson, M., & Frazier, L. (1983). The interaction of syntax and semantics during sentence processing: Eye movements in the analysis of semantically biased sentences. Journal of Verbal Learning and Verbal Behavior, 22(3), 358–374. https://doi.org/10.1016/S0022-5371(83)90236-0 — [doi:10.1016/s0022-5371(83)90236-0](https://doi.org/10.1016/s0022-5371%2883%2990236-0) ↗
  59. Reder, L. M., & Kusbit, G. W. (1991). Locus of the Moses illusion: Imperfect encoding, retrieval, or match? Journal of Memory and Language, 30(4), 385–406. https://doi.org/10.1016/0749-596X(91)90013-A — [doi:10.1016/0749-596x(91)90013-a](https://doi.org/10.1016/0749-596x%2891%2990013-a) ↗
  60. Rodd, J. M., Lopez Cutrin, B., Kirsch, H., Millar, A., & Davis, M. H. (2013). Long-term priming of the meanings of ambiguous words. Journal of Memory and Language, 68(2), 180–198. https://doi.org/10.1016/j.jml.2012.08.002 — [doi:10.1016/j.jml.2012.08.002](https://doi.org/10.1016/j.jml.2012.08.002) ↗
  61. Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021). The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45), e2105646118. https://do… — [doi:10.1073/pnas.2105646118](https://doi.org/10.1073/pnas.2105646118) find
  62. Sidhu, D. M., & Pexman, P. M. (2017). A prime example of the Maluma/Takete effect? Testing for sound symbolic priming. Cognitive Science, 41(7), 1958–1987. https://doi.org/10.1111/cogs.12438 — [doi:10.1111/cogs.12438](https://doi.org/10.1111/cogs.12438) ↗
  63. Sinclair, A., Jumelet, J., Zuidema, W., & Fernández, R. (2022). Structural persistence in language models: Priming as a window into abstract language representations. Transactions of the Association for Computational Linguistics, 10, 1031–1050. https://doi.org/10.1162/tacl_a_00504 ↗
  64. Singer, M., & Spear, J. (2015). Phantom recollection of bridging and elaborative inferences. Discourse Processes, 52(5–6), 356–375. https://doi.org/10.1080/0163853X.2015.1029858 — [doi:10.1080/0163853x.2015.1029858](https://doi.org/10.1080/0163853x.2015.1029858) ↗
  65. Sun, S., Zhang, Y., Yan, J., Gao, Y., Ong, D., Chen, B., & Su, J. (2023, October 16). Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT — A Text-to-SQL Parsing Comparison. Findings of the Association for Computational Linguistics: EMNLP 2023. 11225– 11238.h… find
  66. Van Berkum, J. J. A., van den Brink, D., Tesink, C. M. J. Y., Kos, M., & Hagoort, P. (2008). The neural integration of speaker and message. Journal of Cognitive Neuroscience, 20(4), 580–591. https://doi.org/10.1162/jocn.2008.20054 van Gompel, R. P. G., Pickering, M. J., & Traxler, M. J. (2001). Rean… — [doi:10.1162/jocn.2008.20054](https://doi.org/10.1162/jocn.2008.20054) ↗
  67. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30. https://proceedings.neurips.cc/paper/7181-attention-is-all ↗
  68. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., & Fedus, W. (2022). Emergent abilities of large language models. arXiv. https://doi.org/10.48550/arXiv.2206.07682 ↗
  69. Xu, Z., Jain, S., & Kankanhalli, M. (2024). Hallucination is inevitable: An innate limitation of large language models. arXiv. https://doi.org/10.48550/arXiv.2401.11817 ↗
  70. SUPPLEMENTAL INFORMATION We provided exploratory analyses (preregistered or non-preregistered) here; preregistered exploratory analyses can also be viewed in the preregistrations (osf.io/vu2h3/registrations). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cai-2023-large-language-models-resemble-humans/