Elamur Research
☰
home / papers / 2019 / Multilingual Constituency Parsing with Self-Attention and Pre-Training

Multilingual Constituency Parsing with Self-Attention and Pre-Training

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1340.pdf

Summary

The paper asks how much unsupervised pre-training helps constituency parsing across languages, and whether one parser can serve many languages without a huge parameter cost. The authors extend their earlier self-attentive span-based parser (Kitaev and Klein, 2018). It scores labeled spans with an MLP classifier and decodes with a CKY variant. They feed it fastText, ELMo or BERT representations, then fine-tune on treebanks for English (WSJ), Chinese (Chinese Treebank 5.1) and the nine SPMRL 2013/2014 languages. For ten languages they also train one jointly fine-tuned model. It shares multilingual BERT and the self-attention layers, and adds only a small per-language span classifier. Contextual pre-training improves results for every language, and higher-capacity models do better. The shared model is about ten times smaller than separate per-language parsers and loses little accuracy. Pairwise experiments show that small treebanks can benefit from large ones. The work sets new state-of-the-art results for all 11 languages. This makes it a practical blueprint for compact multilingual parsers, and the code and models are public (self-attentive-parser).

Key points

  • On the WSJ dev set, F1 rises from 93.61 (no pre-training) and 93.72 (fastText) to 95.21 (ELMo), 95.32 (BERT BASE) and 95.70 (BERT LARGE). A 4-model ensemble reaches 95.87.
  • English-only BERT beats multilingual BERT by under 0.3 F1 (95.24 vs. 94.97). Even BERT pre-trained on Chinese Wikipedia roughly matches the treebank-only baseline on English (93.57 F1).
  • The joint ten-language model has 189M parameters, compared with 1,851M for separate models. Its average F1 is 91.12 vs. 91.40, a 3.2% relative error increase, and it actually improves Swedish.
  • The largest treebanks (German, English, Korean) work best as auxiliary languages. Small ones (Swedish, French, Polish) gain the most from joint training. Language pairing is driven more by annotation conventions than by linguistic kinship.
  • Test-set results are 95.8 F1 on WSJ (ensemble) and 91.75 F1 on Chinese Treebank 5.1. On SPMRL, both the joint and per-language models beat all prior systems on every language.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Treebanks

Cited by

156 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 144 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1340 · arXiv 1812.11760

@inproceedings{kitaev2019multilingual,
  title = {{Multilingual Constituency Parsing with Self-Attention and Pre-Training}},
  author = {Kitaev, Nikita and Cao, Steven and Klein, Dan},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1340},
  eprint = {1812.11760},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1340}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in NLP. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995–1005, Jeju Island, Korea.… find
  2. Anders Björkelund, Ozlem Cetinoglu, Agnieszka Faleńska, Richárd Farkas, Thomas Mueller, Wolfgang Seeker, and Zsolt Szántó. 2014. The IMS-Wrocław-Szeged-CIS entry at the SPMRL 2014 shared task: Reranking and morphosyntax meet unlabeled data. In Proceedings of the First Joint Workshop on Statisti… find
  3. Anders Björkelund, Ozlem Cetinoglu, Richárd Farkas, Thomas Mueller, and Wolfgang Seeker. 2013. (Re)ranking meets morphosyntax: State-of-the-art results from the SPMRL 2013 shared task. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 135–145, Se… find
  4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find
  5. Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2331–2336. Association for Computational Linguistics. find
  6. Maximin Coavoux and Benoit Crabbé. 2017. Multilingual lexicalized constituency parsing with word-level auxiliary tasks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 331–336. Association for Computationa… find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018a. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 [cs]. ArXiv: 1810.04805. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018b. BERT: Pre-training of deep bidirectional transformers for language understanding. https: //github.com/google-research/bert/ blob/master/multilingual.md. find
  9. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199–209. Association for Computationa… find
  10. Daniel Fried and Dan Klein. 2018. Policy gradient as a proxy for dynamic oracles in constituency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 469–476. Association for Computational Linguistics. find
  11. Daniel Fried, Mitchell Stern, and Dan Klein. 2017. Improving neural parsing by disentangling model combination and reranking effects. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 161–166. Association for Computational Ling… find
  12. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339. Association for Computational Linguistics. find
  13. Vidur Joshi, Matthew Peters, and Mark Hopkins. 2018. Extending a parser to distant domains using a few dozen partially annotated examples. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1190–1199. Association for Computationa… find our card
  14. Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686. Association for Computational Linguistics. find our card
  15. Jiangming Liu and Yue Zhang. 2017. In-order transition-based constituent parsing. Transactions of the Association for Computational Linguistics, 5:413–424. find
  16. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the 11th Language Resources and Evaluation Conference, Miyazaki, Japan. European Language Resource Association. find our card
  17. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  18. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  19. Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Gal-letebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yanni… find
  20. Zhiyang Teng and Yue Zhang. 2018. Two local models for neural constituent parsing. In Proceedings of the 27th International Conference on Computational Linguistics, pages 119–132. Association for Computational Linguistics. find
  21. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find