Elamur Research⌕
☰
home / papers / 2021 / Frequency Effects on Syntactic Rule Learning in Transformers

Frequency Effects on Syntactic Rule Learning in Transformers

Paper · 2021 · Jason Wei, Dan Garrette, Tal Linzen et al.

Summary

The paper asks whether pre-trained transformers like BERT truly acquire abstract symbols and rules, using English subject–verb agreement (SVA) as a test case. Instead of probing a released checkpoint, the authors pre-train several BERT models from scratch on English Wikipedia. This gives them control over the vocabulary and lets them deliberately change how often specific verb forms appear in the pre-training data. Evaluation uses targeted syntactic evaluation: a verb is masked, and the MLM head must score the correct inflection above the wrong one. Three hypotheses are compared: an idealized symbolic learner, a purely co-occurrence-based item-specific learner, and a symbolic learner with noisy mapping from words to features.

BERT handles subject–verb pairs never seen together in training fairly well, which argues against pure memorization. However, its accuracy is causally affected by both a verb form's absolute frequency and its frequency relative to the competing inflection. Probing classifiers indicate that rare-verb errors come from not learning the verb's number feature. Large frequency imbalances, by contrast, push BERT toward the more common form even against the rule. The study clarifies how training statistics shape apparent rule learning.

Key points

  • Multiple BERT instances were pre-trained on Wikipedia only (2.3 billion tokens), since BooksCorpus is not publicly available.
  • Training their own models let the authors add any verb form to the vocabulary as a single wordpiece. This avoids the restriction to in-vocabulary verbs used by Goldberg (2019).
  • Frequency interventions targeted 60 "verbs of interest" (e.g., emphasize, threaten, propose, reveal) whose inflection frequencies were manipulated in the pre-training corpus.
  • Of the three hypotheses, the "symbolic learner with noisy observations" (H3) fits best. This hypothesis does not fully account for BERT's added sensitivity to relative frequency between competing word forms.
  • The authors describe their study as the first to test these questions through controlled interventions on pre-training data, rather than through post-hoc analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓New York UniversityNew York · United States136 works · 86,390 citations🎓Brown UniversityProvidence · United States58 works · 25,310 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Jason Wei, Dan Garrette, Tal Linzen and Ellie Pavlick. 2021. Frequency Effects on Syntactic Rule Learning in Transformers. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.72

@inproceedings{wei2021frequency,
  title = {{Frequency Effects on Syntactic Rule Learning in Transformers}},
  author = {Jason Wei, Dan Garrette, Tal Linzen and Ellie Pavlick},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.72},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.72}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. 2021. Predicting inductive biases of pre-trained models. In Proc. of ICLR. find
  2. Rui Chaves and Stephanie Richter. 2021. Look at that! BERT can be easily distracted from paying attention to morphosyntax. In Proceedings of the Society for Computation in Linguistics. find
  3. Noam Chomsky. 1956. Three models for the description of language. IRE Transactions on Information Theory, 2(3):113–124. find
  4. Alec Marantz. 2013. Words and rules revisited: Reassessing the role of construction and memory in language. In 27th Pacific Asia Conference on Language, Information, and Computation, PACLIC 2013. National Chengchi University. find
  5. Gabriella Chronis and Katrin Erk. 2020. When is a bishop not like a rook? When it’s like a rabbi! Multi-prototype BERT embeddings for estimating semantic relationships. In Proc. of CoNLL. find
  6. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proc. of EMNLP. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL. our card
  8. Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. 2016. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP. our card
  9. Jerry A. Fodor and Zenon W. Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3–71. find
  10. Yoav Goldberg. 2019. Assessing bert’s syntactic abilities. arXiv preprint arXiv:1901.05287. our card
  11. Emily Goodwin, Koustuv Sinha, and Timothy J. O’Donnell. 2020. Probing linguistic systematicity. In Proc. of ACL. find
  12. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proc. of NAACL. our card
  13. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proc. of NAACL. find
  14. Najoung Kim and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In Proc. of EMNLP. find
  15. Najoung Kim and Paul Smolensky. 2021. Testing for grammatical category abstraction in neural language models. Proceedings of the Society for Computation in Linguistics. find
  16. Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proc. of ICML. find
  17. Tal Linzen and Marco Baroni. 2020. Syntactic structure from deep learning. Annual Review of Linguistics. find
  18. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. TACL. our card
  19. Benjamin Newman, Kai-Siang Ang, Julia Gong, and John Hewitt. 2021. Refining targeted syntactic evaluation of language models. In Proc. of NAACL. find
  20. Judea Pearl. 1988. Probabilistic reasoning in intelligent systems: Networks of plausible inference. find
  21. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. TACL. our card
  22. Paul Smolensky. 1988. On the proper treatment of connectionism. Behavioral and Brain Sciences, 11(1):1–23. find
  23. Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. 2019. Small and practical BERT models for sequence labeling. In Proc. of EMNLP. find
  24. Sara Veldhoen, Dieuwke Hupkes, and Willem Zuidema. 2016. Diagnostic Classifiers: Revealing how Neural Networks Process Hierarchical Structure. In CEUR Workshop Proceedings. find
  25. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP. our card
  26. Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually). In Proc. of EMNLP. find
  27. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019. Can neural networks understand monotonicity reasoning? In Proceedings of the 2019 ACL Workshop BlackboxNLP. find
  28. Charles Yu, Ryan Sie, Nicolas Tedeschi, and Leon Bergen. 2020. Word frequency does not predict grammatical knowledge in language models. In Proc. of EMNLP. find
  29. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proc. of ICCV. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wei-2021-frequency-effects-syntactic-rule-learning/