Elamur Research⌕
☰
home / papers / 2020 / Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank

Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank

Paper · 2020 · Ethan C. Chau, Lucy H. Lin and Noah A. Smith

Summary

Multilingual models such as mBERT serve low-resource language varieties unevenly. Varieties that are barely present in, or absent from, the pretraining data have too little labeled and unlabeled text to train a strong monolingual model. Chau, Lin, and Smith test whether mBERT can be adapted cheaply using only a small unlabeled corpus in the target variety. They propose three methods:

  • Language-adaptive pretraining (LAPT): extra masked-LM training on target-language text.
  • Vocabulary augmentation (VA): fills mBERT's unused vocabulary slots with frequent target-language wordpieces before LAPT.
  • Tiered VA (TVA): the same as VA, but with a higher learning rate for the new embeddings.

They evaluate the methods on UD dependency parsing in Irish, Maltese, Vietnamese and Singlish, using a biaffine parser on top of mBERT layer representations, with frozen and finetuned variants. LAPT reliably beats unadapted mBERT and fastText/ELMo baselines. VA adds smaller but significant gains, while TVA rarely helps beyond VA. The gains are largest for varieties poorly covered by mBERT's pretraining data. This shows that a little unlabeled text can substantially narrow the gap multilingual models show on low-resource languages.

Key points

  • The authors group target varieties by their relation to mBERT's pretraining data:
    • Type 0: Vietnamese, plus Singlish, whose component languages are well represented.
    • Type 1: Irish, which has a tiny Wikipedia.
    • Type 2: Maltese, which mBERT never saw.
  • VA repurposes mBERT's 99 "unused" vocabulary tokens for new wordpieces, so the pretrained architecture is left unchanged. Unlabeled data comes from Wikipedia, with only a 5% sample for Vietnamese, and from the SG Talk Forum for Singlish.
  • For Maltese, LAS rises from 67.06 (frozen mBERT) to 78.51 with LAPT. With finetuning it rises from 76.74 (mBERT+FT) to 83.53 (VA+FT). For Irish, finetuned LAS rises from 72.67 to 76.17 with VA+FT.
  • Maximum relative error reduction over unadapted mBERT tracks language type: about 39% for Maltese, 15% for Irish, 11% for Singlish and 5% for Vietnamese.
  • Code, data splits and hyperparameters are released at github.com/ethch18/parsing-mbert.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States263 works · 180,581 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States219 works · 89,712 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 1,343 worksMultilingual & cross-lingual NLP · 467 worksTreebanks · 118 works

Cite

Cite as Ethan C. Chau, Lucy H. Lin and Noah A. Smith. 2020. Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank. Findings of the Association for Computational Linguistics: EMNLP 2020.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.118

@inproceedings{chau2020parsing,
  title = {{Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank}},
  author = {Ethan C. Chau, Lucy H. Lin and Noah A. Smith},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.118},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.118}
}

References

42 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rodrigo Agerri, Iñaki San Vicente, Jon Ander Campos, Ander Barrena, Xabier Saralegi, Aitor Soroa, and Eneko Agirre. 2020. Give your text representation models some love: the case for Basque. In Proc. of LREC. find
  2. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proc. of Clinical Natural Language Processing Workshop.
  3. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proc. of ACL.
  4. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: Pretrained language model for scientific text. In Proc. of EMNLP. find
  5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL, 5:135–146.
  6. Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multilingual alignment of contextual word representations. In Proc. of ICLR. find
  7. Tuhin Chakrabarty, Christopher Hidey, and Kathy McKeown. 2019. IMHO fine-tuning improves claim detection. In Proc. of NAACL-HLT. find
  8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In Proc. of ACL. find
  9. Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In Proc. of ACL.
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL-HLT.
  11. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proc. of EMNLP-IJCNLP. find
  12. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proc. of ICLR.
  13. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proc. of Workshop for NLP Open Source Software (NLP-OSS).
  14. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proc. of ACL. find
  15. Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised domain adaptation of contextualized embeddings for sequence labeling. In Proc. of EMNLP.
  16. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proc. of ACL. find
  17. Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT: An empirical study. In Proc. of ICLR. find
  18. Dan Kondratyuk and Milan Straka. 2019. 75 languages, 1 model: Parsing Universal Dependencies universally. In Proc. of EMNLP-IJCNLP.
  19. Surafel M. Lakew, Aliia Erofeeva, Matteo Negri, Marcello Federico, and Marco Turchi. 2018. Transfer learning in multilingual neural machine translation with dynamic vocabulary. In Proc. of IWSLT. find
  20. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. In Proc. of NeurIPS. find
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692 [cs.CL]. find
  22. Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity descriptions. In Proc. of ACL. find
  23. David Mueller, Nicholas Andrews, and Mark Dredze. 2020. Sources of transfer in multilingual named entity recognition. In Proc. of ACL. find
  24. Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019a. Low-resource parsing with crosslingual contextualized representations. In Proc. of CoNLL. find
  25. Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019b. Polyglot contextual representations improve crosslingual transfer. In Proc. of NAACL-HLT. find
  26. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL-HLT.
  27. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proc. of ACL.
  28. Samuel Rönnqvist, Jenna Kanerva, Tapio Salakoski, and Filip Ginter. 2019. Is multilingual BERT fluent in language generation? In Proc. of the First NLPL Workshop on Deep Learning for Natural Language Processing. find
  29. Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proc. of NAACL-HLT.
  30. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. of ACL.
  31. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune BERT for text classification? arXiv:1905.05583 [cs.CL].
  32. Peter Trudgill. 2003. A Glossary of Sociolinguistics. Edinburgh University Press. find
  33. Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. 2019. Small and practical BERT models for sequence labeling. In Proc. of EMNLP-IJCNLP. find
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NeurIPS.
  35. Hongmin Wang, Jie Yang, and Yue Zhang. 2019. From genesis to creole language: Transfer learning for singlish universal dependencies parsing and pos tagging. ACM Transactions on Asian and Low-Resource Language Information Processing, 19(1). find
  36. Hongmin Wang, Yue Zhang, GuangYong Leonard Chan, Jie Yang, and Hai Leong Chieu. 2017. Universal dependencies parsing for colloquial Singaporean English. In Proc. of ACL.
  37. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proc. of EMNLP-IJCNLP.
  38. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proc. of the 5th Workshop on Representation Learning for NLP.
  39. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  40. Daniel Zeman, Joakim Nivre, Mitchell Abrams, Noëmi Aepli, Željko Agić, Lars Ahrenberg, Gabrielė Alek-sandravičiūtė, Lene Antonsen, Katya Aplonova, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Miguel B… find
  41. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proc. of EMNLP. find
  42. A Supplementary Material to Accompany Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank This supplement contains further details about the experiments presented in the main paper. A.1 Vocabulary Augmentation and Statistics Language GA MT SING VI Original 12807 49791 3 6955 Augment… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chau-2020-parsing-multilingual-bert-small-corpus/