Elamur Research⌕
☰
home / papers / 2025 / For GPT-4 as with Humans: Information Structure Predicts Acceptability of Long-Distance Dependencies

For GPT-4 as with Humans: Information Structure Predicts Acceptability of Long-Distance Dependencies

Paper · 2025 · Nicole Cuneo, Eleanor Graves, Supantho Rakshit et al.

Summary

Whether large language models grasp subtle form–function links in language is still contested. This paper asks whether GPT-4 reproduces a pattern found in humans: the more backgrounded a constituent is in a plain sentence, the less acceptable it is as the target of a long-distance dependency (LDD), such as a wh-question, a relative clause or a cleft. This idea is the Backgrounded Constructions are Islands (BCI) account. The authors gave GPT-4 the same zero-shot Likert tasks used with human participants. One task measured backgroundedness through a negation test. The other collected acceptability ratings on LDD versions of the sentences and on the base sentences.

GPT-4's backgroundedness judgments predicted its LDD ratings but had little effect on its base-sentence ratings, which is the interaction seen in people. Its responses also tracked the human data. The result held on fresh stimuli, which addresses data contamination. A third study raised a constituent's prominence in a context sentence, and acceptability of the matching wh-question went up, which points to a causal effect. The work suggests that island effects can be partly explained by information structure, and that LLMs can serve as tools for probing linguistic theory.

Key points

  • The human benchmark came from a large study with 144 base sentences covering many construction types: various adjuncts, manner-of-speaking verb complements, double objects, to-datives, main and relative clauses, and parasitic and non-parasitic gaps. Each base sentence was hand-converted into wh-Qs, discourse-linked questions and relative clauses.
  • Backgroundedness was measured with a negation task. The model was told to assume a negated sentence was true, then asked how likely it was that the embedded content still held. Content that survives negation is treated as more backgrounded.
  • Study 1a used the original stimuli and replicated the human interaction: backgroundedness predicted LDD ratings but not declarative ratings.
  • Study 1b used new stimuli to reduce contamination risk. It also added a fourth LDD type, it-clefts, which had not been tested on humans, and found converging results.
  • Study 2 manipulated emphasis in a preceding context sentence, and wh-question acceptability rose for the emphasized constituent. All probing was done on GPT-4 (Jan 2025) with zero-shot, minimal prompts, and requests were throttled to avoid caching artifacts.

Interesting ideas

  • I1. Use negation as a graded probe for presupposition: negate the whole sentence and ask whether a sub-part still holds, so that backgroundedness becomes a number rather than a yes/no label.
  • I2. Predict an interaction rather than a single effect: the predictor should change judgments on one sentence type (LDDs) but not on a matched control (base declaratives), so general noise or plausibility cannot explain the result.
  • I3. Test causality by editing the context instead of only correlating: boost a constituent's prominence in a preceding sentence and see whether later judgments shift.
  • I4. Guard against memorization by rebuilding stimuli from scratch and adding a construction type the original benchmark never covered (here it-clefts).
  • I5. Explain extraction acceptability through information structure rather than pure syntactic island rules, so it becomes a gradient score tied to how backgrounded the source phrase is.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag could tag each clause or adjunct with a backgroundedness score by checking what survives sentential negation, which would make presupposition and at-issue content explicit in frames.
  • I2 (learn): evaluations could pair each test with a control condition where the feature should not matter, which works as a built-in negative control.
  • I3 (world): the text world could track discourse prominence of entities across sentences and test whether emphasis changes which referents are preferred later, for example in coreference or question focus.
  • I4 (learn): when an LLM teacher or a test set is drawn from published material, add freshly written items and an unseen construction to check that agreement is not recall.
  • I5 (tree): when the parser has to attach a fronted wh-phrase or relativized element, a backgroundedness prior on candidate host clauses (e.g., causal "by" adjuncts over temporal "while" adjuncts) could rank gap sites in an explainable way.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksNeuroscience · 186 worksCognitive Neuroscience · 185 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Nicole Cuneo, Eleanor Graves, Supantho Rakshit and Adele E. Goldberg. 2025. For GPT-4 as with Humans: Information Structure Predicts Acceptability of Long-Distance Dependencies. arXiv:2505.09005.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.09005 · arXiv 2505.09005

@misc{cuneo2025gpt,
  title = {{For GPT-4 as with Humans: Information Structure Predicts Acceptability of Long-Distance Dependencies}},
  author = {Nicole Cuneo, Eleanor Graves, Supantho Rakshit and Adele E. Goldberg},
  year = {2025},
  doi = {10.48550/arxiv.2505.09005},
  eprint = {2505.09005},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.09005}
}

References

37 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ambridge, B., & Goldberg, A. E. (2008). The island status of clausal complements: Evidence in favor of an information structure explanation. Cognitive Linguistics 19(3), 349-381. ↗
  2. Cuneo, N., & Goldberg, A. (2022). Islands effects without extraction: The discourse functions of constructions predict island status. In Proc. of the Cog. Sci Soc: 44:44. find
  3. Cuneo, N., & Goldberg, A. E. (2023). The discourse functions of grammatical constructions explain an enduring syntactic puzzle. Cognition, 240, 105563. ↗
  4. Basmov, V., Goldberg, Y., & Tsarfaty, R. (2023). Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds. arXiv preprint arXiv:2305.14785. https://arxiv.org/pdf/2305.14785 ↗
  5. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021, March). On the dangers of stochastic parrots: Can language models be too big? 🦜 . In Proc. of 2021 ACM conference on fairness, accountability, and transparency (pp. 610-623). find
  6. Chomsky, N. (1973). Conditions on transformations. A Festschrift for Morris Halle/Holt, Rinehart, and Winston. find
  7. Grimshaw, J. (1986). Subjacency and the S/S′ Parameter. Linguistic Inquiry, 364-369. find
  8. Dąbrowska, E. (2013). Functional constraints, usage, and mental grammars: A study of speakers' intuitions about questions with long-distance dependencies. Cognitive Linguistics 24(4), 633-665. Dentella, V., Günther, F., Murphy, E., Marcus, G. and Leivada, E. “Testing AI on Language Comprehension Tasks Reveals Insensitivity to Underlying Meaning.” Scientific Reports 14, no. 1 (November 14, 2024): 28083. https://doi.org/10.1038/s41598-024-79531-8. ↗
  9. Dentella, V., Günther, F., & Leivada, E. (2023). Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response bias. PNAS 120(51), e2309583120. ↗
  10. Erteschik-Shir, N. (1973). On the nature of island constraints (PhD thesis, MIT). find
  11. Fergus, A., Belluck, A., Cuneo, N. and Goldberg, A. E. (2025). Islands result from clash of functions: Single-conjunct wh-Qs. Proc. of the Cog.Sci. Society. find
  12. Francis, E. J., & Michaelis, L. A. (2017). When relative clause extraposition is the right choice, it’s easier. Language and Cognition, 9(2), 332-370. ↗
  13. Gauthier, J., Hu, J., Wilcox, E., Qian, P., & Levy, R. (2020, July). SyntaxGym: An online platform for targeted evaluation of language models. In Proc. of the 58th Annual Meeting of the ACL Givón, T. (Ed.). (2020). Discourse and syntax. Brill.
  14. Goldberg, A. E. (2006). Constructions at Work: the nature of generalization in language. OUP: Oxford. find
  15. Goldberg, A. E. (2013) Backgrounded constituents cannot be “extracted”. In J. Sprouse (Ed). Syntactic Islands. OUP: Oxford. 221-238. ↗
  16. Halliday, M. A. K. (2013). Halliday in the 21st Century: Volume 11. A&C Black. find
  17. Hofmeister, P., & Sag, I. A. (2010). Cognitive constraints and island effects. Language, 86(2), 366. Howitt, K., Sathvik N., Dods, A., and Hopkins, R. M. “Generalizations across Filler-Gap Dependencies in Neural Language Models.” arXiv, October 23, 2024. http://arxiv.org/abs/2410.18225. ↗
  18. Hu, J., & Levy, R. (2023). Prompting is not a substitute for probability measurements in large language models. arXiv preprint arXiv:2305.13264. ↗
  19. Hu, J., Mahowald, K., Lupyan, G., Ivanova, A., & Levy, R. (2024). Language models align with human judgments on key grammatical constructions. PNAS. 121(36), e2400917121. ↗
  20. Jeretic, P., Warstadt, A., Bhooshan, S., & Williams, A. (2020). Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. arXiv preprint arXiv:2004.03066. ↗
  21. Kallini, J., Papadimitriou, I., Futrell, R., Mahowald, K, and Potts, C. 2024. “Mission: Impossible Language Models.” http://arxiv.org/abs/2401.06416. ↗
  22. Karttunen, L. (1974). Presupposition and Linguistic Context. Theoretical Linguistics, 1(1-2), 182-194. ↗
  23. Kobzeva, A., Arehalli, S., Linzen, T., & Kush, D. (2023, June). Neural Networks Can Learn Patterns of Island-insensitivity in Norwegian. In Proc. of the Soc. for Computation in Linguistics 2023 (pp. 175-185). ↗
  24. Lambrecht, K. (1994). Information structure and sentence form: Topic, focus, and the mental representations of discourse referents. Cambridge University Press. find
  25. Liu, Y., Winckel, E., Abeillé, A., Hemforth, B., & Gibson, E. (2022). Structural, functional, and processing perspectives on linguistic island effects. Annual Review of Linguistics, 8(1), 495-525. ↗
  26. Lu, J., Pan, D., & Degen, J. (2024). Evidence for a discourse account of manner-of-speaking islands. LSA presentation. ↗
  27. Mandelkern, M., & Linzen, T. (2024). Do Language Models’ Words Refer? Computational Linguistics, 1-10. Mao, R., Hemforth, B., and Abeille e, A. (2024) The discourse function explains “islands”. AMLaP. Online hal-04859279 Mulders, I., & Ruys, E. G. (2024). ChatGPT as an informant. Nota Bene, 1(2), 242-260. ↗
  28. Namboodiripad, S., Cuneo, N., Kramer, M. A., Sedarous, Y., Bisnath, F., Sugimoto, Y., & Goldberg, A. E. (2022). Backgroundedness predicts island status of non-finite adjuncts in English. Proc. of the Cog. Sci. Soc. find
  29. Phillips, C. (2006). The real-time status of island phenomena. Language, 795-823. ↗
  30. Potts, C. (2004). The logic of conventional implicatures. OUP. ↗
  31. Potts, C. (2023). Characterizing English Preposing in PP constructions. LingBuzz lingbuzz/007495. ↗
  32. Ross, J. R. (1967). Constraints on variables in syntax. Ph.D. Thesis, Massachusetts Institute of Technology. find
  33. Sieker, J., & Zarrieß, S. (2023, December). When Your Language Model Cannot Even Do Determiners Right: Probing for Anti-Presuppositions and the Maximize Presupposition! Principle. In Proc. of the 6th blackboxnlp workshop: Analyzing and interpreting neural networks for NLP (pp. 180-198).
  34. Sprouse, J. (2013) (Ed). Experimental syntax and island effects. Cambridge University Press. ↗
  35. Sravanthi, S. L., Doshi, M., Kalyan, T. P., Murthy, R., Bhattacharyya, P., & Dabre, R. (2024). PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities. arXiv preprint arXiv:2401.07078. Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, E., Wang, S-F., and Bowman, S.R. “BLiMP: The Benchmark of Linguistic Minimal Pairs for English.” Transactions of the ACL 8 (December 2020): 377–92. //doi.org/10.1162/tacl_a_00321. ↗
  36. Ward, G., & Birner, B. J. (2019). 12 Discourse effects of word order variation. Semantics: Sentence and Information Structure, 413. ↗
  37. Winckel, E., Abeillé, A., Hemforth, B., & Gibson, E. (2025). Discourse-based constraints on long-distance dependencies generalize across constructions in English and French. Cognition, 254, 105950. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cuneo-2025-gpt-4-humans-information-structure/