Elamur Research⌕
☰
home / papers / 2023 / Construction Grammar Provides Unique Insight into Neural Language Models

Construction Grammar Provides Unique Insight into Neural Language Models

Summary

This is a position paper. The authors ask whether large pretrained language models (PLMs) learn constructions in the sense of Construction Grammar (CxG). A construction here is a pairing of form and meaning that may have open slots and cannot be reduced to the sum of its individual words. The authors explain why this question matters both for the theory of language modeling and for applied tasks. For example, machine translation often loses the meaning of unusual constructions. They then review general probing methods, such as comparing contextual embeddings and minimal pairs. Their conclusion is that these methods touch on only some aspects of constructional knowledge. Separately, the authors analyze the few studies designed specifically for CxG. The paper presents no new data or experiments. Its contribution is a research agenda: a dedicated probing methodology, annotated data, and a shared multilingual resource of constructions are needed. It is also worth testing whether the transformer architecture itself and the training objective hinder the learning of constructions. For NLP, this is a guide to evaluating models through more than just the lens of generative grammar.

Key points

  • An example of failure in an applied task: DeepL translates the sentence "She sneezed the foam off her cappuccino" (the caused-motion construction, CMC) into German, Italian and Turkish in such a way that the meaning "to blow the foam off by sneezing" is lost. In the Russian translation the motion is conveyed, but not the fact that the sneeze caused it.
  • The main examples of constructions in the paper are the Comparative Correlative ("The X-er the Y-er") and the levels of constructions according to Goldberg (2013): from individual words to fully unfilled templates such as the ditransitive or the passive.
  • The authors propose creating shared online access to existing constructicons and a unified formal representation of the form of constructions. The model for this should be the experience of Universal Dependencies (UD) in popularizing dependency syntax.
  • Two hypotheses about fundamental difficulties. First: the non-compositional meaning of a construction is not tied to specific tokens, so it can be stored only in the model's weights. Second: for a masked or causal LM objective, the meaning of a construction is often simply not needed to predict a word correctly.
  • Possible ways to improve: masking entire constructions at once (Tseng et al., 2022), grouping sentences with the same construction into "documents" for pretraining (Tayyar Madabushi et al., 2020), adding something like a positional embedding or a control token to the architecture.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Language models & pretraining

Cite

Cite as Leonie Weissweiler, Taiqi He, Naoki Otani, David R. Mortensen, Lori Levin and Hinrich Schütze. 2023. Construction Grammar Provides Unique Insight into Neural Language Models. Proceedings of the First International Workshop on Construction Grammars and NLP (CxGs+NLP, GURT/SyntaxFest 2023).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2302.02178

@inproceedings{weissweiler2023construction,
  title = {{Construction Grammar Provides Unique Insight into Neural Language Models}},
  author = {Leonie Weissweiler, Taiqi He, Naoki Otani, David R. Mortensen, Lori Levin and Hinrich Schütze},
  booktitle = {Proceedings of the First International Workshop on Construction Grammars and NLP (CxGs+NLP, GURT/SyntaxFest 2023)},
  year = {2023},
  eprint = {2302.02178},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.02178}
}

References

39 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discou… our card
  2. Adele E.. Goldberg. 1995. Constructions: A construction grammar approach to argument structure. University of Chicago Press. find
  3. Adele E. Goldberg. 2013. 1415 Constructionist Approaches. In The Oxford Handbook of Construction Grammar. Oxford University Press. find
  4. Giulia ML Bencini and Adele E Goldberg. 2000. The contribution of argument structure constructions to sentence meaning. Journal of Memory and Language, 43(4):640–651. find
  5. Yoko Hasegawa, Russell Lee-Goldman, Kyoko Hirose Ohara, Seiko Fujii, and Charles J Fillmore. 2010. On expressing measurement and comparison in english and japanese. Contrastive studies in construction grammar, 10. find
  6. H. C. Boas and I. A. Sag. 2012. Sign-Based Construction Grammar. Center for the Study of Language and Information. find
  7. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129… our card
  8. Gabriella Chronis and Katrin Erk. 2020. When is a bishop not like a rook? when it’s like a rabbi! multi-prototype BERT embeddings for estimating semantic relationships. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 227–244, Online. Association for Computatio… find
  9. William Croft. 2001. Radical construction grammar: Syntactic theory in typological perspective. Oxford University Press on Demand. find
  10. Taelin Karidi, Yichu Zhou, Nathan Schneider, Omri Abend, and Vivek Srikumar. 2021. Putting words in BERT’s mouth: Navigating contextualized vector spaces with pseudowords. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10300–10313, Online and Punta C… find
  11. William Croft. 2022. Morphosyntax: Constructions of the World’s Languages. Cambridge University Press. find
  12. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308. our card
  13. Paul Kay and Ivan A. Sag. 2012. Cleaning up the big mess: Discontinuous dependencies and complex determiners. In H. C. Boas and I. A. Sag, editors, Sign-Based Construction Grammar. Center for the Study of Language and Information. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  15. Bai Li, Zining Zhu, Guillaume Thomas, Frank Rudzicz, and Yang Xu. 2022. Neural reality of argument structure constructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7410–7423, Dublin, Ireland. Association for Computatio… find
  16. Jonathan Dunn. 2017. Computational learning of construction grammars. Language and cognition, 9(2):254–292. find
  17. Emmy Liu and Graham Neubig. 2022. Are representations built from the ground up? an empirical examination of local composition in language models. arXiv preprint arXiv:2210.03575. find
  18. C. J. Fillmore, R. Lee-Goldman, and R. Rhodes. 2012. The framenet constructicon. In H. C. Boas and I. A. Sag, editors, Sign-Based Construction Grammar. Center for the Study of Language and Information. find
  19. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  20. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021. Probing for idiomaticity in vector space models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3551–3564, Online. As… our card
  21. Benjamin Lyngfelt, Lars Borin, Kyoko Ohara, and Tiago Timponi Torrent. 2018. Constructicography: Constructicon development across languages, volume 22. John Benjamins Publishing Company. find
  22. Adele Goldberg. 2006. Constructions at work: The nature of generalization in language. Oxford University Press, Oxford, UK. find
  23. Louis Martin, Éric de la Clergerie, Benoît Sagot, and Antoine Bordes. 2020. Controllable sentence simplification. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4689– 4698, Marseille, France. European Language Resources Association. linguistic generalizations (even… find
  24. James D McCawley. 1988. The comparative conditional construction in english, german, and chinese. In Annual Meeting of the Berkeley Linguistics Society, volume 14, pages 176–187. find
  25. Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021. Frequency effects on syntactic rule learning in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 932–948, Online and Punta Cana, Dominican Republic. Association for Computation… find
  26. Miriam R. L. Petruck and Gerard de Melo, editors. 2014. Proceedings of Frame Semantics in NLP: A Workshop in Honor of Chuck Fillmore (1929-2014). Association for Computational Linguistics, Baltimore, MD, USA. find
  27. Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, and Hinrich Schütze. 2022. The better your syntax, the better your semantics? probing pretrained language models for the English comparative correlative. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing… find
  28. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  29. Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. our card
  30. Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019. Does bert make any sense? interpretable word sense disambiguation with contextualized embeddings. arXiv preprint arXiv:1909.10430. — [arXiv:1909.10430](https://arxiv.org/abs/1909.10430) find
  31. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. our card
  32. Yadollah Yaghoobzadeh, Katharina Kann, T. J. Hazen, Eneko Agirre, and Hinrich Schütze. 2019. Probing for semantic classes: Diagnosing the meaning content of word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5740–5753, Florence, Italy.… find
  33. Lane Schwartz, Coleman Haley, and Francis Tyers. 2022. How to encode arbitrarily complex morphology in word embeddings, no corpus needed. In Proceedings of the first workshop on NLP applications to field linguistics, pages 64–76, Gyeongju, Republic of Korea. International Conference on Computational… find
  34. Weidong Zhan. 2017. On theoretical issues in building a knowledge database of chinese constructions. Journal of Chinese Information Processing, 31(1):230–238. find
  35. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615. our card
  36. Alexander Ziem, Alexander Willich, and Sascha Michel. forthcoming. Constructing constructicons. John Benjamins Publishing Company. find
  37. Harish Tayyar Madabushi, Laurence Romain, Dagmar Divjak, and Petar Milin. 2020. CxGBERT: BERT meets construction grammar. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4020–4032, Barcelona, Spain (Online). International Committee on Computational Linguistics… our card
  38. Yu-Hsiang Tseng, Cing-Fang Shih, Pin-Er Chen, Hsin-Yu Chou, Mao-Chang Ku, and Shu-Kai Hsieh. 2022. CxLM: A construction and context-aware language model. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6361– 6369, Marseille, France. European Language Resources As… find
  39. Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020. Learning which features matter: RoBERTa acquires a preference for find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/weissweiler-2023-construction-grammar-provides-unique-insight/