Elamur Research
☰
home / papers / 2019 / A Structural Probe for Finding Syntax in Word Representations

A Structural Probe for Finding Syntax in Word Representations

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-1419.pdf

Summary

Probing studies had shown that contextual word vectors carry pieces of syntactic information. They had not tested whether a whole dependency tree can be read from these vectors. Hewitt and Manning propose the structural probe, which learns a single linear map applied to a model's word vectors. After the map, the squared Euclidean distance between two words should equal the number of edges between them in the parse tree. A second variant uses the squared vector norm to predict how deep each word sits in the tree. The probe is trained on Penn Treebank sentences annotated with Stanford Dependencies, and parses are rebuilt by taking a minimum spanning tree over the predicted distances. Representations from ELMo, BERT-base and BERT-large recover trees far better than non-contextual or randomly contextualized baselines. Only a low-dimensional subspace is needed. This suggests that pretrained language models encode syntax geometrically, even though they never saw parse trees during training. The method gives a cheap and easy-to-interpret way to inspect the vector spaces of other models and languages.

Key points

  • On the PTB WSJ test set, the best distance probe reaches 82.5 UUAS (BERT-large, layer 15). ELMo layer 1 scores 77.0, while the strongest baseline, a randomly initialized BiLSTM over character embeddings (PROJ0), gets 59.8.
  • The depth probe on BERT-large layer 16 finds the root word 90.1% of the time, with a Spearman correlation of 0.89 against true tree depth.
  • Across models, BERT-large beats BERT-base, which beats ELMo. Syntax is concentrated in the middle layers rather than the first or last ones.
  • Making the linear map larger than about 64–128 dimensions brings no further gain, and all three models need roughly the same rank.
  • The code is released at github.com/john-hewitt/structural-probes. The authors note that the probe tests a deliberately strict hypothesis: syntax as a global distance structure across the whole vector space.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

631 papers of our library cite it (of the 140673 whose bibliographies we read); 29 of them have a page here, 602 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1419

@inproceedings{hewitt2019structural,
  title = {{A Structural Probe for Finding Syntax in Word Representations}},
  author = {Hewitt, John and Manning, Christopher D.},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1419},
  url = {https://doi.org/10.18653/v1/n19-1419}
}

References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representations. find our card
  2. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861–872. Association for Com… find our card
  3. Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. Deep RNNs encode soft hierarchical syntax. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 14–19. Association for Computational Linguistics. find
  4. Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single \\$&!\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo… find our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  6. Richard Futrell, Ethan Wilcox, Takashi Morita, and Roger Levy. 2018. RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency. arXiv preprint arXiv:1809.01329. find
  7. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… find our card
  8. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  9. Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… find
  10. Tal Linzen. 2018. What can linguistics and deep learning contribute to each other? arXiv preprint arXiv:1809.04179. — [arXiv:1809.04179](https://arxiv.org/abs/1809.04179) find
  11. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521– 535. find our card
  12. Tal Linzen and Brian Leonard. 2018. Distinct patterns of syntactic agreement errors in recurrent networks and humans. In Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 692– 697. Cognitive Science Society, Austin, TX. find
  13. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational linguistics, 19(2):313–330. find our card
  14. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In LREC. find our card
  15. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pa… find
  16. William L Hamilton, Rex Ying, and Jure Leskovec. 2017. Representation learning on graphs: Methods and applications. arXiv preprint arXiv:1709.05584. — [arXiv:1709.05584](https://arxiv.org/abs/1709.05584) find
  17. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Pau… find our card
  18. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907–926. find
  19. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop. find
  20. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… find
  21. Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499–1509. find our card
  22. Marten van Schijndel and Tal Linzen. 2018. Modeling garden path effects without explicit hierarchical syntax. In Tim Rogers, Marina Rau, Jerry Zhu, and Chuck Kalish, editors, Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 2600–2605. Cognitive Science Society, Austi… find
  23. Gongbo Tang, Mathias Müller, Annette Rios, and Rico Sennrich. 2018. Why self-attention? A targeted evaluation of neural machine translation architectures. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4263–4272. Association for Computational Lingui… find
  24. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Lear… find our card