Elamur Research
☰
home / papers / 2018 / Constituency Parsing with a Self-Attentive Encoder

Constituency Parsing with a Self-Attentive Encoder

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1805.01052

Summary

This paper asks whether a strong constituency parser needs a recurrent (LSTM) encoder. The authors keep the chart-based span decoder of Stern et al. (2017a), with changes from Gaddy et al. (2018), and replace the encoder with a stack of self-attention layers adapted from the Transformer of Vaswani et al. (2017). Attention shows how information moves between words, so the authors use it to study the model. They find it relies mostly on word position rather than word content. Keeping content and position signals in separate channels, which they call a "factored" encoder, improves accuracy. They also compare several word-level inputs: external POS tags, character embeddings of prefixes and suffixes, a character LSTM, and pre-trained ELMo vectors. On the Penn Treebank WSJ test set, the best model without external data scores 93.55 F1. With ELMo it scores 95.13 F1, and both were the best single-model results at the time. The parser also beats earlier published scores on 8 of 9 SPMRL languages. The work shows that attention-based encoders can match or beat RNNs for syntactic parsing and are easier to inspect.

Key points

  • On the WSJ development set, the self-attentive encoder (8 layers, 8 heads) scores 92.67 F1, compared with 92.24 for the LSTM encoder of Gaddy et al. (2018) using the same decoder.
  • Development F1 rises at each step: factored attention gives 93.15, adding a CharLSTM gives 93.61, and adding ELMo gives 95.21. The ELMo result came from a smaller 4-layer encoder.
  • WSJ test results are 93.55 F1 for the CharLSTM model using WSJ data only and 95.13 F1 for the ELMo model.
  • On SPMRL 2013/2014, the parser sets new best results for 8 of 9 languages. Swedish is the exception, where an ensemble with reranking (Björkelund et al., 2014) stays ahead. ELMo was tested only on English.
  • Training the WSJ-only model took about 18 hours on one Tesla K80 GPU. Code and English models are published at github.com/nikitakit/self-attentive-parser.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

283 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 272 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1805.01052 · arXiv 1805.01052

@misc{kitaev2018constituency,
  title = {{Constituency Parsing with a Self-Attentive Encoder}},
  author = {Kitaev, Nikita and Klein, Dan},
  year = {2018},
  doi = {10.48550/arxiv.1805.01052},
  eprint = {1805.01052},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1805.01052}
}

References

19 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer Normalization. arXiv:1607.06450 [cs, stat]. ArXiv: 1607.06450. find our card
  2. Anders Björkelund, Ozlem Cetinoglu, Agnieszka Faleńska, Richárd Farkas, Thomas Mueller, Wolfgang Seeker, and Zsolt Szántó. 2014. The IMS-Wrocław-Szeged-CIS entry at the SPMRL 2014 shared task: Reranking and morphosyntax meet unlabeled data. In Proceedings of the First Joint Workshop on Statisti… find
  3. Anders Björkelund, Ozlem Cetinoglu, Richárd Farkas, Thomas Mueller, and Wolfgang Seeker. 2013. (Re)ranking meets morphosyntax: State-of-the-art results from the SPMRL 2013 shared task. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 135–145, Se… find
  4. Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2331–2336. Association for Computational Linguistics. find
  5. Maximin Coavoux and Benoit Crabbé. 2017. Multilingual lexicalized constituency parsing with word-level auxiliary tasks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 331–336. Association for Computationa… find
  6. James Cross and Liang Huang. 2016. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1–11. Association for Computational Linguistics. find
  7. Greg Durrett and Dan Klein. 2015. Neural CRF parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 302–312. Association for Computational Linguistics… find
  8. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199–209. Association for Computationa… find
  9. Daniel Fried, Mitchell Stern, and Dan Klein. 2017. Improving neural parsing by disentangling model combination and reranking effects. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 161–166. Association for Computational Ling… find
  10. David Gaddy, Mitchell Stern, and Dan Klein. 2018. What’s going on in neural constituency parsers? An analysis. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. find
  11. David Hall, Greg Durrett, and Dan Klein. 2014. Less grammar, more features. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 228–237. find
  12. Jiangming Liu and Yue Zhang. 2017. In-order transition-based constituent parsing. Transactions of the Association for Computational Linguistics, 5:413–424. find
  13. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… find
  14. Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Gal-letebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yanni… find
  15. Mitchell Stern, Jacob Andreas, and Dan Klein. 2017a. A minimal span-based neural constituency parser. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 818–827. Association for Computational Linguistics. find our card
  16. Mitchell Stern, Daniel Fried, and Dan Klein. 2017b. Effective inference for generative neural parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1695–1700. Association for Computational Linguistics. find
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Process… find
  18. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in Neural Information Processing Systems 28, pages 2755– 2763. Curran Associates, Inc. find
  19. Symbol Description N N d model h d k d v d f f find