Elamur Research⌕
☰
home / papers / 2020 / CxGBERT: BERT meets Construction Grammar

CxGBERT: BERT meets Construction Grammar

Summary

The authors test whether BERT has access to information that Construction Grammar (CxG) considers distinct from lexical semantics. This concerns learned form–meaning pairings, ranging from idioms to fully abstract syntactic schemas. The study is built around two questions. First: does BERT change if it is explicitly given knowledge about constructions? To this end, the CxGBERT model was pre-trained on sentences grouped by a shared construction. The constructions were extracted using a modified version of the method of Dunn (2017). CxGBERT was compared with BERT Base Clone, trained on Wikipedia articles, and with the control model BERT Random. No noticeable difference was found. Hence, such information is probably already present in BERT, or something functionally equivalent substitutes for it. The second question: how well does BERT recognize constructions? After fine-tuning, the model reliably distinguished constructions, especially narrow and semantically specific ones. With frozen weights, the results were weak. The authors conclude that this knowledge is stored in BERT's internal layers, and that in language itself constructions are largely duplicated by lexical-semantic cues. This is important for understanding what neural networks extract from text. In addition, BERT can be used for the automated compilation of constructicons.

Key points

  • Three models were compared: CxGBERT (sentences grouped by constructions), BERT Base Clone (Wikipedia), and BERT Random. BERT Random consistently performs worse, so the gain comes specifically from grouping sentences in the training data.
  • The subset of constructions each of which has fewer than 10,000 example sentences contains over 21,000 constructions. On it, BERT reaches about 85% accuracy, even though it was fine-tuned on only 500 examples. That is not even enough for every construction to receive at least one example.
  • The model most easily recognizes specific patterns with few examples, such as Personal Pronoun + didn't + V + how. General patterns like Noun + -s or Preposition + his are harder for it. The authors doubt that such general patterns are constructions in the CxG sense at all.
  • Limitation: the extracted constructions are mostly short and obvious (for example, MODAL + BE + Past-Participle), and this may partly explain the high accuracy.
  • The code, hyperparameters, and pre-trained models are available on GitHub: H-TayyarMadabushi/CxGBERT-BERT-meets-Construction-Grammar.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

Cite as Harish Tayyar Madabushi, Laurence Romain, Dagmar Divjak and Petar Milin. 2020. CxGBERT: BERT meets Construction Grammar. Proceedings of the 28th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.coling-main.355 · arXiv 2011.04134

@inproceedings{madabushi2020cxgbert,
  title = {{CxGBERT: BERT meets Construction Grammar}},
  author = {Harish Tayyar Madabushi, Laurence Romain, Dagmar Divjak and Petar Milin},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.355},
  eprint = {2011.04134},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.coling-main.355}
}

References

38 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open… our card
  2. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861–872, Vancouver, Canada,… our card
  3. Ethan A. Chi, John Hewitt, and Christopher D. Manning. 2020. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5564–5577, Online, July. Association for Computational Linguistics. find
  4. Noam Chomsky. 1957. Syntactic Structures. Mouton and Co., The Hague. find
  5. Grzegorz Chrupała and Afra Alishahi. 2019. Correlating neural and symbolic representations of language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2952–2962, Florence, Italy, July. Association for Computational Linguistics. find
  6. Alexis Conneau, German Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… our card
  7. William Croft and D. Alan Cruse. 2004. Cognitive Linguistics. Cambridge Textbooks in Linguistics. Cambridge University Press. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. our card
  9. Computational learning of construction grammars. Language and Cognition, Rui Feng, Congcong Yang, and Yunhua Qu. 2020. A word embedding model for analyzing patterns and their distributional semantics. Journal of Quantitative Linguistics, 0(0):1–26. find
  10. A.E. Goldberg. 1995. Constructions: A Construction Grammar Approach to Argument Structure. Cognitive Theory of Language and Culture Series. University of Chicago Press. find
  11. Adele E. Goldberg. 2002. Surface generalizations: An alternative to alternations. Cognitive Linguistics, 13(4):327 – 356. find
  12. A.E. Goldberg. 2006. Constructions at Work: The Nature of Generalization in Language. Oxford linguistics. Oxford University Press. find
  13. Micah B. Goldwater, Marc T. Tomlinson, Catharine H. Echols, and Bradley C. Love. 2011. Structural priming as structure-mapping: Children use analogies from previous utterances to guide sentence production. Cognitive science, 35 1:156–70. find
  14. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… our card
  15. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zh… our card
  16. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129… our card
  17. M. Hilpert. 2014. Construction Grammar and Its Application to English. Edinburgh University Press Series. Edinburgh University Press. find
  18. S. Hunston and G. Francis. 2000. Pattern Grammar: A Corpus-driven Approach to the Lexical Grammar of English. Pattern Grammar: A Corpus-driven Approach to the Lexical Grammar of English. John Benjamins Publishing Company. find
  19. Daniel Jurafsky. 1992. An on-line computational model of human sentence interpretation. In Proceedings of the Tenth National Conference on Artificial Intelligence, AAAI’92, page 302–308. AAAI Press. find
  20. Michael P Kaschak and Arthur M Glenberg. 2000. Constructing meaning: The role of affordances and grammatical constructions in sentence comprehension. Journal of Memory and Language, 43(3):508 – 529. find
  21. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4(0):521–535. our card
  22. Tal Linzen. 2018. What can linguistics and deep learning contribute to each other? CoRR, abs/1809.04179. — [arXiv:1809.04179](https://arxiv.org/abs/1809.04179) find
  23. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec… our card
  24. Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019b. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pa… find
  25. B. Lyngfelt, L. Borin, K. Ohara, and T.T. Torrent. 2018. Constructicography: Constructicon development across languages. Constructional Approaches to Language. John Benjamins Publishing Company. find
  26. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. CoRR, abs/1609.07843. our card
  27. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. CoRR, abs/1802.05365. our card
  28. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, November. Association for Computational Lin… our card
  30. Kyle Richardson, Hai Hu, Lawrence S Moss, and Ashish Sabharwal. 2020. Probing natural language inference models through semantic fragments. In AAAI, pages 8713–8721. our card
  31. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in bertology: What we know about how BERT works. CoRR, abs/2002.12327. our card
  32. J.M.H. Sinclair. 1987. Looking Up: An Account of the COBUILD Project in Lexical Computing and the Development of the Collins COBUILD English Language Dictionary. Collins Cobuild Dictionaries Series. Collins ELT. find
  33. Anatol Stefanowitsch and Stefan Th. Gries. 2003. Collostructions: Investigating the interaction of words and constructions. International Journal of Corpus Linguistics, 8(2):209–243. find
  34. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. CoRR, abs/1905.05950. our card
  35. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Lea… our card
  36. Nai-Lung Tsao and David Wible. 2013. Word similarity using constructions as contextual features. In Proceedings of the Joint Symposium on Semantic Processing. Textual Inference and Structures in Corpora, pages 51–59, Trento, Italy, November. find
  37. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… our card
  38. Pei Zhou, Rahul Khanna, Bill Yuchen Lin, Daniel Ho, Xiang Ren, and Jay Pujara. 2020. Can BERT reason? logically equivalent probes for evaluating the inference capabilities of language models. CoRR, abs/2005.00782. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/madabushi-2020-cxgbert-bert-meets-construction-grammar/