Elamur Research
☰
home / papers / 2019 / Open Sesame: Getting inside BERT's Linguistic Knowledge

Open Sesame: Getting inside BERT's Linguistic Knowledge

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W19-4825.pdf

Summary

BERT's contextual representations improve many downstream tasks, but it is unclear whether they capture sentence structure or mostly word order. The authors examine pre-trained bert-base-uncased and bert-large-uncased in two ways.

First, they train single-token perceptron probes (diagnostic classifiers) using a "poverty of the stimulus" design. The training data come from context-free grammars and can be solved by either a linear rule (pick the first auxiliary or noun) or a structural rule. A harder generalization set then separates the two. A separate n th-token probe on Penn Treebank WSJ sentences measures positional information.

Second, they propose a confusion score. It measures how much distractor noun phrases pull BERT's self-attention away from the true subject in subject-verb agreement, or from the true antecedent of a reflexive anaphor.

Both analyses show that position is well encoded in the lower layers and gives way to hierarchical encoding higher up. Attention reacts to the type of syntactic embedding and to number and gender matching, but only partially. It lacks the sharp structural sensitivity humans show with reflexives. These findings help explain BERT's strength on structure-heavy tasks.

Key points

  • Models: bbu stacks 12 layers, uses 12 heads for attention, and has a hidden size of 768. blu has 24 layers, 16 heads and width 1024. Confusion scores were computed with bbu only.
  • Synthetic data for the main-auxiliary and subject-noun tasks contain 40000 training, 10000 development and 10000 generalization examples. The generalization sets add relative clauses (auxiliary task) or compound and possessive nouns (subject task).
  • The n th-token task uses WSJ sentences of 10–30 BERT tokens, split 21142/3017/2999, with 2 ≤ n ≤ 9.
  • Mean confusion scores in the unambiguous agreement conditions (A1–A4) are near the uniform baseline of 1 rather than the ideal 0. PP distractors confuse BERT more than relative-clause distractors, and confusion decreases with layer depth. Positional information fades between layers 3 and 4, and confusion rises at layer 4.
  • Code is released at github.com/yongjie-lin/bert-opensesame and builds on Vig's bertviz library.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

142 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 136 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4825

@inproceedings{lin2019open,
  title = {{Open Sesame: Getting inside BERT's Linguistic Knowledge}},
  author = {Lin, Yongjie and Tan, Yi Chern and Frank, Robert},
  booktitle = {Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP},
  year = {2019},
  doi = {10.18653/v1/w19-4825},
  url = {https://doi.org/10.18653/v1/w19-4825}
}

References

31 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference for Learning Representations. find our card
  3. Kathryn Bock and J. Cooper Cutting. 1992. Regulating mental energy: Performance units in language production. Journal of Memory and Language, 31(1):99–127. find
  4. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and diagnostic classifiers reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907926. find
  5. Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. 2019. Activation atlas. https://distill.pub/2019/ activation-atlas [Accessed: 19 Apr 2019]. ↗
  6. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. Computing Research Repository, abs/1506.02078. find our card
  7. Noam Chomsky. 1980. Rules and representations. Behavioral and Brain Sciences, 3(1):1–15. find
  8. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Computing Research Repository, abs/1412.6980. find
  9. Michael Collins. 2002. Discriminative training methods for hidden Markov models: Theory and experiments with perceptron algorithms. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing, pages 1–8. Association for Computational Linguistics. find our card
  10. Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. The emergence of number and syntax units in LSTM language models. Computing Research Repository, abs/1903.07435. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. Computing Research Repository, abs/1810.04805. find
  12. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521– 535. find our card
  13. Brian Dillon, Alan Mishler, Shayne Sloggett, and Colin Phillips. 2013. Contrasting intrusion profiles for agreement and anaphora: Experimental and modeling evidence. Journal of Memory and Language, 69(2):85–103. find
  14. Tal Linzen and Brian Leonard. 2018. Distinct patterns of syntactic agreement errors in recurrent networks and humans. In Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 690–695. find
  15. Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neura… find
  16. Yang Liu. 2019. Fine-tune BERT for extractive summarization. In Computing Research Repository, volume abs/1903.10318. — [arXiv:1903.10318](https://arxiv.org/abs/1903.10318) find
  17. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. find
  18. Yoav Goldberg. 2019. Assessing BERT’s syntactic abilities. Computing Research Repository, abs/1901.05287. find our card
  19. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… find our card
  20. R Thomas McCoy, Robert Frank, and Tal Linzen. 2018. Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks. In Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 2093–2098. find
  21. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  22. John Hewitt and Christopher Manning. 2019. A structural probe for finding syntax in word representations. In International Conference for Learning Representations. find our card
  23. Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. 2018. The building blocks of interpretability. https://distill.pub/2018/ building-blocks [Accessed: 19 Apr 2019]. ↗
  24. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543. find
  25. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. Computing Research Repository, abs/1802.05365. find
  26. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. https://d4mucfpksywv.cloudfront. net/better-language-models/ language-models.pdf [Accessed: 19 Apr 2019]. ↗
  27. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Re… find our card
  28. Jesse Vig. 2019. Visualizing attention in transformer-based language models. Computing Research Repository, abs/1904.02679. — [arXiv:1904.02679](https://arxiv.org/abs/1904.02679) find
  29. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264–1274, Melbourne, Australia. Associati… find our card
  30. Thomas Wolf. 2019. Some additional experiments extending the tech report “Assessing BERT’s Syntactic Abilities” by Yoav Goldberg. https: //huggingface.co/bert-syntax/ extending-bert-syntax.pdf [Accessed: 19 Apr 2019]. find our card
  31. Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2018. Recent trends in deep learning based natural language processing. IEEE Computational Intelligence Magazine, 13(3):55–75. find