Elamur Research⌕
☰
home / papers / 2024 / A Construction Grammar Corpus of Varying Schematicity: A Dataset for the Evaluation of Abstractions in Language Models

A Construction Grammar Corpus of Varying Schematicity: A Dataset for the Evaluation of Abstractions in Language Models

Summary

The authors test whether large language models (LLMs) acquire abstract grammatical generalizations. To do so, they draw on Construction Grammar (CxG). The key concept of the paper is schematicity. It indicates whether the slots of a construction require specific fixed words or can be filled by any element with the required semantic role. The authors compiled the CoGS (Construction Grammar Schematicity) corpus: more than 500 sentences illustrating 10 English constructions. The constructions are ordered from lexically fixed (Let-alone) to fully schematic argument constructions (Resultative). Collecting examples was difficult because existing corpora lack CxG-style annotation. On this corpus, GPT-3.5 and GPT-4 were probed with several types of prompts, including prompts with an example sentence for each construction. The result is the same for both models and across all prompt formulations: the more schematic the construction, the worse the models recognize it. They practically fail to notice the shared features of purely schematic constructions. Thus, the capacity of LLMs for abstraction is limited. The corpus makes it possible to compare human grammar with the way LLMs model language.

Key points

  • CoGS contains 10 constructions, grouped by level of abstraction. Lexically fixed: Let-alone, Much-less. Partially fixed: Way-manner, Comparative-Correlative. Mostly schematic, with a fixed preposition "at" or "with": Conative, Causative-with. Fully schematic: Caused-Motion, Intransitive-Motion, Ditransitive, Resultative.
  • Accuracy with the best prompt for the levels "fixed / partially schematic / fully schematic": GPT-3.5 — 84.00 / 75.17 / 54.00 %, GPT-4 — 98.34 / 92.67 / 62.33 %.
  • The models tested had at least 175 billion parameters. The drop in accuracy with increasing schematicity is consistent across models and prompts.
  • Each construction has a description of its meaning and a description of its form. The form is described with Hoffmann (2022) templates at two levels: phonological (fixed words) and morphosyntactic (schematic slots).
  • The corpus, prompts and model responses are openly published on GitHub: H-TayyarMadabushi/Construction_Grammar_Schematicity_Corpus-CoGS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Language models & pretraining

Cite

Cite as Claire Bonial and Harish Tayyar Madabushi. 2024. A Construction Grammar Corpus of Varying Schematicity: A Dataset for the Evaluation of Abstractions in Language Models. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bonial2024construction,
  title = {{A Construction Grammar Corpus of Varying Schematicity: A Dataset for the Evaluation of Abstractions in Language Models}},
  author = {Claire Bonial and Harish Tayyar Madabushi},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  url = {https://aclanthology.org/2024.lrec-main.22/}
}

References

35 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discou… our card
  2. Claire Bonial, Bianca Badarau, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Tim O’Gorman, Martha Palmer, and Nathan Schneider. 2018. Abstract meaning representation of constructions: The more we include, the better the representation. In Proceedings of the Eleventh International Conference on Languag… find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… our card
  4. Bastian Bunzeck and Sina Zarrieß. 2023. Entrenchment matters: Investigating positional and constructional sensitivity in small and large language models. In Proceedings of the 2023 CLASP Conference on Learning with Small Data (LSD), pages 25–37. find
  5. Jonathan Dunn. 2023. Exploring the constructicon: Linguistic analysis of a computational cxg. arXiv preprint arXiv:2301.12642. find
  6. Charles Fillmore, Paul Kay, and Mary O’Connor. 1988. Regularity and idiomaticity in grammatical constructions: The case of let alone. Language, 64(3):501–531. find
  7. Charles J Fillmore, Russell Lee-Goldman, and Russell Rhodes. 2012. The framenet constructicon. Sign-based construction grammar, (193):309– 372. find
  8. Adele E Goldberg. 1995. Constructions: A construction grammar approach to argument structure. University of Chicago Press. find
  9. Adele E Goldberg. 2003. Constructions: A new theoretical approach to language. Trends in cognitive sciences, 7(5):219–224. find
  10. Adele E. Goldberg. to appear. A chat about constructionist approaches and lmms. Constructions and Frames. find
  11. Thomas Hoffmann. 2022. Construction Grammar. Cambridge Textbooks in Linguistics. Cambridge University Press. find
  12. Thomas Hoffmann and Graeme Trousdale. 2013. The Oxford handbook of construction grammar. Oxford University Press. find
  13. Jennifer Hu and Roger Levy. 2023. Prompting is not a substitute for probability measurements in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5040–5060, Singapore. Association for Computational Linguistics. our card
  14. Ray Jackendoff. 2010. Meaning and the lexicon: the parallel architecture 1975-2010. OUP Oxford. find
  15. Ray S Jackendoff. 1992. Semantic structures, volume 18. MIT press. find
  16. Joan Bybee. 2006. From usage to grammar: The mind’s response to repetition. Language, pages 711–733. find
  17. Joan Bybee. 2010. Language, usage and cognition. Cambridge University Press. find
  18. Noam Chomsky. 2014. The minimalist program. find
  19. Matt A Johnson and Adele E Goldberg. 2013. Evidence for automatic accessing of constructional meaning: Jabberwocky sentences prime associated verbs. Language and Cognitive Processes, 28(10):1439–1452. find
  20. Michael P Kaschak and Arthur M Glenberg. 2000. Constructing meaning: The role of affordances and grammatical constructions in sentence comprehension. Journal of memory and language, 43(3):508–529. find
  21. Harish Tayyar Madabushi, Laurence Romain, Dagmar Divjak, and Petar Milin. 2020. Cxgbert: Bert meets construction grammar. arXiv preprint arXiv:2011.04134. our card
  22. Harish Tayyar Madabushi, Laurence Romain, Petar Milin, and Dagmar Divjak. 2024. Construction grammar and language models. In Mirjam Fried and Kiki Nikiforidou, editors, The Cambridge Handbook of Construction Grammar. Accepted for publication, to appear. find
  23. Kevin Knight et al. 2020. Abstract Meaning Representation (AMR) Annotation Release 3.0 LDC2020T02. Web Download. Philadelphia: Linguistic Data Consortium. find
  24. Natalia Levshina and Kris Heylen. 2014. A radically data-driven construction grammar: Experiments with dutch causative constructions. Extending the scope of Construction Grammar, 54:17. find
  25. Michael Tomasello. 2005. Constructing a language: A usage-based theory of language acquisition. Harvard university press. find
  26. Bai Li, Zining Zhu, Guillaume Thomas, Frank Rudzicz, and Yang Xu. 2022. Neural reality of argument structure constructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7410–7423, Dublin, Ireland. Association for Computatio… find
  27. Michael Tomasello. 2006. Construction grammar for kids. Constructions. find
  28. Tim Veenboer and Jelke Bloem. 2023. Using collostructional analysis to evaluate bert’s representation of linguistic constructions. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12937–12951. find
  29. Zhuang Liu, Wayne Lin, Ya Shi, and Jun Zhao. 2021. A robustly optimized bert pre-training approach with post-training. In China National Conference on Chinese Computational Linguistics, pages 471–484. Springer. our card
  30. Alex Warstadt, Yian Zhang, Haau-Sing Li, Haokun Liu, and Samuel R Bowman. 2020. Learning which features matter: Roberta acquires a preference for linguistic generalizations (eventually). arXiv preprint arXiv:2010.05358. find
  31. Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. 2023. Are emergent abilities in large language models just in-context learning? find
  32. Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, and Hinrich Schütze. 2022. The better your syntax, the better your semantics? probing pretrained language models for the english comparative correlative. arXiv preprint arXiv:2210.13181. find
  33. Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought in large language models: a cognitive perspective. find
  34. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational linguistics, 31(1):71–106. our card
  35. Florent Perek. 2018. Recent change in the productivity and schematicity of the way-construction: A distributional semantic analysis. Corpus Linguistics and Linguistic Theory, 14(1):65–97. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bonial-2024-construction-grammar-corpus-varying-schematicity/