Elamur Research⌕
☰
home / papers / 2019 / Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing

Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing

Paper · 2019 · Yuxuan Wang, Wanxiang Che, Jiang Guo et al.

Summary

Zero-shot cross-lingual dependency parsing needs lexical features that carry over between languages. Earlier transfer parsers relied on static cross-lingual embeddings, and training cross-lingual language models such as XLM from scratch requires large parallel corpora and heavy compute. The authors propose Cross-Lingual BERT Transformation (CLBT), an off-line method that maps contextualized embeddings of target languages from multilingual BERT (mBERT) into the space of English BERT (enBERT). Tokens in 10,000 Europarl sentence pairs per language are aligned with fast_align at the WordPiece level. A linear map is then fitted on the aligned token vectors, either as a closed-form orthogonal SVD solution or through unconstrained gradient descent (GD). The mapped embeddings feed a Biaffine graph-based parser that is trained only on the English UD v2.2 treebank and applied directly to 17 target languages. CLBT raises averaged LAS by 2.91 points over the previous static-embedding state of the art. It also rivals XLM on shared languages while using a small fraction of its parallel data. This makes adding a new language pair cheap, typically within hours.

Key points

  • Setup: English is the source and 17 languages are targets, all drawn from UD v2.2 and limited to languages with available Europarl data. The baselines are FT-SVD (FastText plus SVD, the previous state of the art), raw mBERT, and XLM (XNLI-15 model).
  • Main results: Averaged target LAS is 60.63 for FT-SVD, 60.53 for mBERT, 63.09 for CLBT (SVD), and 63.54 for CLBT (GD). Combining CLBT with FT-SVD adds a further 0.33 (GD) or 0.51 (SVD) LAS.
  • Comparison with XLM: CLBT beats XLM on French, Spanish and Bulgarian but trails it on German. XLM used 0.2M–13.1M parallel sentences per language, whereas CLBT used 10,000 (about 0.4M tokens).
  • Training data size: With SVD, most of the 13 tested languages peak at about 5,000 parallel sentences. For Germanic and Romance languages, just 100 sentences give near-comparable LAS.
  • Analysis and code: A t-SNE analysis on SENSEVAL2 data for nature/naturaleza shows that the transformation brings same-sense tokens across languages closer together. Code is available at github.com/WangYuxuan93/CLBT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States159 works · 65,157 citations🎓Harbin Institute of TechnologyHarbin · China76 works · 27,918 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States54 works · 29,441 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Multilingual & cross-lingual NLP · 467 works Dependency parsing · 120 works

Cite

Cite as Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu and Ting Liu. 2019. Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1575

@inproceedings{wang2019cross,
  title = {{Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing}},
  author = {Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu and Ting Liu},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1575},
  url = {https://doi.org/10.18653/v1/d19-1575}
}

References

35 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wasi Uddin Ahmad, Zhisong Zhang, Xuezhe Ma, Eduard H. Hovy, Kai-Wei Chang, and Nanyun Peng. 2018. Near or far, wide range zero-shot cross-lingual dependency parsing. CoRR, abs/1811.00570. find
  2. David Alvarez-Melis and Tommi Jaakkola. 2018. Gromov-Wasserstein alignment of word embedding spaces. In Proc. of EMNLP, pages 1881–1890. our card
  3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. our card
  4. Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. Towards better ud parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation. In Proc. of CoNLL, pages 55–64. Association for Computational Linguistics. find
  5. Xilun Chen and Claire Cardie. 2018. Unsupervised multilingual word embeddings. In Proc. of EMNLP, pages 261–270. Association for Computational Linguistics. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. our card
  7. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proc. of ICLR. our card
  8. Chris Dyer, Adam Lopez, Juri Ganitkevitch, Jonathan Weese, Ferhan Ture, Phil Blunsom, Hendra Setiawan, Vladimir Eidelman, and Philip Resnik. 2010. cdec: A decoder, alignment, and learning framework for finite-state and context-free translation models. In Proc. of the ACL 2010 System Demonstrations,… our card
  9. Philip Edmonds and Scott Cotton. 2001. Senseval-2: Overview. In The Proc. of the Second International Workshop on Evaluating Word Sense Disambiguation Systems, SENSEVAL ’01, pages 1–5, Stroudsburg, PA, USA. Association for Computational Linguistics. our card
  10. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2015. Cross-lingual dependency parsing based on distributed representations. In Proc. of ACL and IJCNLP, pages 1234–1244. our card
  11. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2016. A representation learning framework for multi-source transfer parsing. In Proc. of AAAI, pages 2734–2740. find
  12. Suzana Ilić, Edison Marrese-Taylor, Jorge Balazs, and Yutaka Matsuo. 2018. Deep contextualized word representations for detecting sarcasm and irony. In Proc. of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 2–7. Association for Computationa… our card
  13. Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of ICLR. find
  14. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proc. of COLING, pages 1459–1474. our card
  15. Tomáš Kočiský, Karl Moritz Hermann, and Phil Blunsom. 2014. Learning bilingual word representations by marginalizing alignments. In Proc. of ACL, pages 224–229. Association for Computational Linguistics. find
  16. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proc. of the tenth Machine Translation Summit, pages 79–86, Phuket, Thailand. AAMT. find
  17. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. CoRR, abs/1901.07291. find
  18. Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Herv Jgou. 2018. Word translation without parallel data. In Proc. of ICLR. find
  19. Angeliki Lazaridou, Georgiana Dinu, and Marco Baroni. 2015. Hubness and pollution: Delving into cross-space mapping for zero-shot learning. In Proc. of ACL and IJCNLP, pages 270–280. Association for Computational Linguistics. our card
  20. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems 30, pages 6294–6305. Curran Associates, Inc. our card
  21. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal dependency annotation for multilingual parsing. In Proc. of ACL, pag… our card
  22. Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proc. of EMNLP, pages 62–72. our card
  23. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. CoRR, abs/1309.4168. find
  24. Phoebe Mulcaire, Jungo Kasai, and Noah Smith. 2019. Polyglot contextual representations improve crosslingual transfer. In Proc. of NAACL. find
  25. Joakim Nivre, Mitchell Abrams, Željko Agić, and et al. 2018. Universal dependencies 2.2. LIN-DAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  26. Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proc. of ACL, pages 1756–1765. Association for Computational Linguistics. our card
  27. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL, pages 2227–2237. Association for Computational Linguistics. our card
  28. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proc. of ACL, pages 4996–5001. our card
  29. Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2018. Cross-lingual transfer learning for multilingual task oriented dialog. CoRR, abs/1810.13327. our card
  30. Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proc. of NAACL, pages 1599–1613. our card
  31. Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proc. of ICLR. our card
  32. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of bert. In Proc. of EMNLP. our card
  33. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  34. Min Xiao and Yuhong Guo. 2014. Distributed word representation learning for cross-lingual dependency parsing. In Proc. of CoNLL, pages 119–129. Association for Computational Linguistics. find
  35. Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proc. of NAACL, pages 1006–1011. Association for Computational Linguistics. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2019-cross-lingual-bert-transformation-zero/