Elamur Research⌕
☰
home / papers / 2024 / Non-native speakers of English or ChatGPT: Who thinks better?

Non-native speakers of English or ChatGPT: Who thinks better?

Paper · 2024 · Mohammed Q. Shormani

Summary

The paper asks whether advanced non-native speakers of English (NNSs) or ChatGPT handle hard syntax better. It pushes back on claims, notably by Piantadosi (2023), that large language models can serve as theories of language. The test case is center-embedding: relative clauses nested inside one another, as in triple or deeper constructions where subjects pile up before their verbs. These sentences are known to strain human working memory.

Fifteen advanced English learners took part: fourteen were writing MA or PhD theses and one was a level-four undergraduate. They received a single center-embedded sentence with a question about it. ChatGPT (version 3.5) got the same item. About three quarters of the humans answered correctly. ChatGPT failed both at interpreting the sentence and at judging it.

The author reads human errors as performance limits caused by memory load, not missing grammatical knowledge. ChatGPT's errors, in contrast, are taken to reflect its statistically learned representations. The conclusion is that LLMs are useful engineering tools but weak models of human linguistic cognition.

Key points

  • There were 15 NNS participants, all advanced learners of English: 14 graduate (MA/PhD) thesis writers and 1 fourth-level student.
  • The stimulus was one center-embedding English sentence, given to both the humans and ChatGPT-3.5.
  • NNS accuracy was 73.3%, and ChatGPT erred on both the processing (interpretation) task and the judgement task.
  • The background examples come from Karlsson (2007). They include a triple-embedded relative with "that" and a harder quadruple reduced-relative version with no relative pronouns.
  • The author lists these limitations: only one construction type was tested, and only ChatGPT-3.5 was used. Data are available from the author on request.

Interesting ideas

  • I1. Center-embedded relatives work as a targeted stress test, because a system must link each stacked subject to its own verb across long, nested distances.
  • I2. The paper separates two tasks: interpreting a hard sentence and judging whether it is well-formed. A system can fail one, the other, or both.
  • I3. The competence-versus-performance distinction says failures under memory load differ from failures caused by missing rules.
  • I4. Comprehension of nested clauses can be probed with a "who did what to whom" question rather than by asking for a parse.
  • I5. Reduced relatives without relative pronouns are flagged as markedly harder than versions with "that", since the overt marker gives the reader a cue.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Build a small graded test set of single, double and triple center-embedded sentences, with and without "that", to check whether the perceptron parser and gparse attach each nsubj and acl:relcl correctly as depth grows.
  • I2 (learn): Evaluate grammaticality judgement and meaning extraction as separate scores, so that a parser accepting a sentence is not mistaken for a correct understanding of it.
  • I3 (learn): Label errors on deep embeddings as either resource-bound (beam or window limits) or rule-bound (missing attachment logic), so repair rules go after the right cause.
  • I4 (frames): Pair each center-embedded test sentence with agent and patient questions for every verb, and check that the frame builder assigns the right roles to each stacked noun.
  • I5 (answer): Include pronoun-less nested relatives in reading-comprehension tests, to measure how much the answering readers depend on overt markers like "that".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ibb UniversityIbb · Yemen6 works · 13 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksNeuroscience · 165 worksCognitive Neuroscience · 164 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Mohammed Q. Shormani. 2024. Non-native speakers of English or ChatGPT: Who thinks better?. arXiv:2412.00457 · F1000Research, 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.12688/f1000research.161306.2 · arXiv 2412.00457

@misc{shormani2024non,
  title = {{Non-native speakers of English or ChatGPT: Who thinks better?}},
  author = {Mohammed Q. Shormani},
  year = {2024},
  doi = {10.12688/f1000research.161306.2},
  eprint = {2412.00457},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.12688/f1000research.161306.2}
}

References

47 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ambridge, B., & Blything, L. (2024). Large language models are better than theoretical linguists at theoretical linguistics. Theoretical Linguistics, 50(1-2), 33-48. ↗
  2. Arbib, M. A. (1982). From artificial intelligence to neurolinguistics. Neural models of language processes, 77-94. ↗
  3. Arbib, M. A., & Caplan, D. (1979). Neurolinguistics must be computational. Behavioral and Brain Sciences, 2(3), 449-460. ↗
  4. Berwick, R. & Chomsky, N. (2016). Why only us: Language and evolution. Cambridge: MIT Press. find
  5. Chomsky, N. (1995). The Minimalist Program. Cambridge, MAA MIT Press. find
  6. Chomsky, N. & Miller, G. A. (1963). Introduction to the formal analysis of natural languages. In Luce, R. D. Bush, R. R. & Galanter, E. (eds), Handbook of Mathematical Psychology, 2. John Wiley, New York, NY, 269-321. find
  7. Chomsky, N. (1957). Syntactic structures. The Hague: Mouton. find
  8. Chomsky, N. (1965). Aspects of the theory of syntax. Cambridge: MIT Press. ↗
  9. Chomsky, N. (1975). The logical structure of linguistic theory. University of Chicago. Illinois: Chicago. find
  10. Chomsky, N. (2001). The architecture of language. Oxford University Press. find
  11. Chomsky, N. (2005). Three factors in language design. Linguistic Inquiry, 36, 1-22. ↗
  12. Chomsky, N. (2008). On phases. In Otero, C. Freidin, R. & Zubizaretta, M. (eds.), Foundational Issues in Linguistic Theory. Cambridge: MAA MIT Press. 133– 166. find
  13. Chomsky, N. (2009). Cartesian linguistics: A chapter in the history of rationalist thought. Cambridge University Press. find
  14. Chomsky, N. (2021). Minimalism: Where are we now, and where can we hope to go. Gengo Kenkyu (Journal of the Linguistic Society of Japan), 160, 1-41. find
  15. Dentella, V., Günther, F., Murphy, E., Marcus, G., & Leivada, E. (2024). Testing AI on language comprehension tasks reveals insensitivity to underlying meaning. Scientific Reports, 14(1), 28083. ↗
  16. Cook, V. (1983). Chomsky’s universal grammar and second language learning. Applied Linguistics, 6(1), 2-18. find
  17. Dickey, M. (1995). Constraints on the sentence processor and the distribution of resumptive pronouns. Technical report, University of Massachesetts-Amherst. find
  18. Ettinger, A. (2020). What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8, 34-48. ↗
  19. Frazier, L. & Rayner, K. (1988). Parameterizing the language processing system: left-vs. right-branching within and across languages. In Hawkins, J. A. (ed.), Explaining language universals. Oxford: Blackwell. 246–279. find
  20. Frazier, L. (1985). Syntactic complexity. In Dowty, D. & Karttunen A., Zwicky, L. (eds.), Natural Language Processing: Psychological, Computational and Theoretical Perspectives. Cambridge: Cambridge University Press. find
  21. Gibson, E. (1998). Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1), 1-76. ↗
  22. Haenlein, M., Kaplan, A. (2019). A brief history of artificial intelligence: On the past, present, and future of artificial intelligence. California management review, 61(4), 5-14. ↗
  23. Harris, N., Butani, A., & Hashmy, S. (2024). Enhancing embedding performance through large language model-based text enrichment and rewriting. arXiv preprint arXiv:2404.12283. ↗
  24. Hauser, M. D., Chomsky, N., & Fitch, W. T. (2002). The faculty of language: What is it, who has it, and how did it evolve?. Science, 298(5598), 1569-1579. ↗
  25. Jenkins, L. (2000). Biolinguistics: Exploring the biology of language. Cambridge: Cambridge University Press. find
  26. Jiang, T., Huang, S., Luan, Z., Wang, D., & Zhuang, F. (2023). Scaling sentence embeddings with large language models. arXiv preprint arXiv:2307.16645. ↗
  27. Karlsson, F. (2007). Constraints on multiple center-embedding of clauses. Journal of Linguistics, 43(2), 365-392. ↗
  28. Karlsson, F. (2010). Working memory constraints on multiple center-embedding. In Proceedings of the Annual Meeting of the Cognitive Science Society. 32, 32. find
  29. Katz, Y. (2012). Noam Chomsky on where artificial intelligence went wrong. https://www.theatlantic.com/technology/archive/2012/11/noam-chomsky-on-where-artificial-intelligence-went-wrong/261637/ ↗
  30. Katzir, R. (2023). Why large language models are poor theories of human linguistic cognition. A reply to Piantadosi (2023). Manuscript. Tel Aviv University. https://lingbuzz.net/lingbuzz/007190 . Kaufer, D. S. (1979). The competence/performance distinction in linguistic theory. Philosophy of the social sciences, 9(3), 257-275. ↗
  31. Kodner, J., Payne, S., & Heinz, J. (2023). Why linguistics will thrive in the 21st century: A reply to Piantadosi (2023). arXiv preprint arXiv:2308.03228. ↗
  32. McCarthy, J., Minsky, M. L., Rochester, N., & Shannon, C. E. (1955). A proposal for the Dartmouth summer research project on artificial intelligence, AI Magazine, 27(4), 12. find
  33. Nelson, R. J. (1978). The competence-performance distinction in mental philosophy. Synthese, 337-381. ↗
  34. Piantadosi, S. T. (2023). Modern language models refute Chomsky’s approach to language. From fieldwork to linguistic theory, 353. ↗
  35. Shormani, M. (2013). An introduction to English syntax: A generative approach. Lambert Academic Publishing, Germany. find
  36. Shormani, M. (2014a). The nature of language acquisition: Where L1 and L2 acquisition meet? Journal of Literature, Languages and Linguistic, 4. 24-34. find
  37. Shormani, M. (2014b). Collocability difficulty: a UG-based model for stable acquisition. Journal of Literature, Languages and Linguistics, 4, 54-64. find
  38. Shormani, M. (2016). Biolinguistics, the ‘magnetic’ mechanism of Language Faculty and language acquisition. Journal of Teaching and Teacher Education, 4(01), 71-88. ↗
  39. Shormani, M. (2017). SVO, (silent) topics and the interpretation of referential pro: A discourse-syntax interface approach. Italian Journal of Linguistics, 29(2), 91-159. ↗
  40. Shormani, M. (2023). L2 acquisition of Wh-interrogatives at the syntax-discourse interface: interface hypothesis again. F1000Research, 12. ↗
  41. Shormani, M. (2024a). Generative linguistics contribution to artificial intelligence: Where this contribution lies? https://doi.org/10.48550/arXiv.2410.20221 ↗
  42. Shormani, M. (2024b). Introducing Minimalism: A Parametric Variation. Lincom Europa Press. find
  43. Thomas, J. D. (1995). Center-embedding and self-embedding in human language processing (Unpublished PhD Dissertation, MIT). find
  44. Turing, A. M. (1950). Computing machinery and intelligence. Mind, New Series, 59(236). 433-460. find
  45. Uehara, K. & Bradley, D. C. (2002). Center-embedding problem and the contribution of nominative case repetition. In Nakayama, M. (ed.), Sentence Processing in East Asian Languages, Staonford, California: CSLI Publications, 257-287. find
  46. White, L. (2003). Second language acquisition and universal grammar. Cambridge: Cambridge University Press. ↗
  47. Zhong, Q., Ding, L., Liu, J., Du, B., & Tao, D. (2023). Can ChatGPT understand too? A comparative study on ChatGPT and fine-tuned BERT. arXivpreprint arXiv:2302.10198. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shormani-2024-non-native-speakers-english-chatgpt/