Elamur Research⌕
☰
home / papers / 2022 / Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation

Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation

Paper · 2022 · Xinyi Wang, Sebastian Ruder and Graham Neubig

Summary

Multilingual pretrained models such as mBERT work poorly for languages that had little or no text during pretraining. Standard fixes, like continued MLM training or translate-train, need monolingual or parallel corpora, and most of the world's languages lack them. The authors turn to bilingual lexicons, which exist for far more languages. Using PanLex entries, they translate English sentences word by word to create synthetic target-language data in two ways:

  • Pseudo MLM: pseudo-monolingual text for continued pretraining.
  • Pseudo Trans-train: pseudo-labeled task data for fine-tuning.

Words missing from the lexicon are left in English. The authors also test ways to clean this noisy data, such as label distillation and using a little parallel text. They evaluate two scenarios: No-Text, with essentially no target text, and Few-Text, with a small amount of monolingual data. Across 19 under-represented languages and 3 tasks, the synthetic data helps consistently. The best strategy depends on how much real text is available. The work suggests a cheap way to extend NLP to thousands of underserved languages.

Key points

  • Estimated share of the world's ~7,000 languages covered by each resource: mBERT about 1%, Wikipedia/CommonCrawl 4%, the Bible 23%, and lexicons roughly 70%.
  • Reported improvements reach up to 15 points without extra monolingual text and up to 5 points when such text is added.
  • The No-Text and Few-Text settings correspond to Joshi et al.'s "Left-Behinds" and "Scraping-Bys" groups. Together these make up about 85% of languages.
  • Experiments use English as the only source language and are limited to Latin-script targets. Morphology and word-order differences are ignored because analyzers are lacking for these languages.
  • Code and data are released on GitHub (cindyxinyiwang/expand-via-lexicon-based-adaptation). PanLex lists over 5,000 languages, though some have only a few words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Xinyi Wang, Sebastian Ruder and Graham Neubig. 2022. Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.61

@inproceedings{wang2022expanding,
  title = {{Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation}},
  author = {Xinyi Wang, Sebastian Ruder and Graham Neubig},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.61},
  url = {https://doi.org/10.18653/v1/2022.acl-long.61}
}

References

38 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. natius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa…
  2. Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of EMNLP 2020.
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In ACL. find
  4. Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In NeurIPS. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL.
  6. Dan Kondratyuk and Milan Straka. 2019. 75 languages, 1 model: Parsing universal dependencies universally. In EMNLP, Hong Kong, China.
  7. Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird, and Trevor Cohn. 2016. Learning crosslingual word embeddings without bilingual corpora. arXiv preprint arXiv:1606.09403. find
  8. Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In EMNLP, Online. Association for Computational Linguistics.
  9. Abteen Ebrahimi and Katharina Kann. 2021. How to adapt your pretrained multilingual model to 1600 languages. In ACL, Online. Association for Computational Linguistics.
  10. Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017. Cheap translation for cross-lingual named entity recognition. In EMNLP, Copenhagen, Denmark. Association for Computational Linguistics.
  11. Jost Gippert, Nikolaus Himmelmann, Ulrike Mosel, et al. 2006. Essentials of language documentation. Mouton de Gruyter Berlín. find
  12. Stephan Gouws and Anders Søgaard. 2015. Simple task-specific bilingual word embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1386–1390, Denver, Colorado. Association for Computational… find
  13. Emily Kennedy Gref. 2016. Publishing in North American Indigenous Languages. Ph.D. thesis, University of London. find
  14. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In ACL, Online. find
  15. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168. find
  16. Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In NAACL, Online.
  17. Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, et al. 2018. Universal dependencies 2.2. find
  18. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of ACL 2018. find
  19. Junjie Hu, Melvin Johnson, Orhan Firat, Aditya Siddhant, and Graham Neubig. 2021. Explicit Alignment Objectives for Multilingual Bidirectional Encoders. In Proceedings of NAACL 2021. find
  20. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. In ICML.
  21. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In ACL, Online. Association for Computational Linguistics.
  22. Yash Khemchandani, Sarvesh Mehtani, Vaidehi Patil, Abhijeet Awasthi, Partha Talukdar, and Sunita Sarawagi. 2021. Exploiting language relatedness for low web-resource language model adaptation: An Indic languages study. In ACL, Online. Association for Computational Linguistics. find
  23. Arya D. McCarthy, Rachel Wicks, Dylan Lewis, Aaron Mueller, Winston Wu, Oliver Adams, Garrett Nicolai, Matt Post, and David Yarowsky. 2020. The Johns Hopkins University Bible corpus: 1600+ tongues for typological exploration. In LREC, pages 2884–2892, Marseille, France. European Language Resources A…
  24. Robert Östling and Jörg Tiedemann. 2016. Efficient word alignment with Markov Chain Monte Carlo. Prague Bulletin of Mathematical Linguistics, 106:125–146. find
  25. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
  26. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In ACL, pages 1946–1958, Vancouver, Canada. ACL.
  27. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In EMNLP, Online. Association for Computational Linguistics.
  28. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2021. UNKs Everywhere: Adapting Multilingual Language Models to New Scripts. In Proceedings of EMNLP 2021.
  29. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In ACL, Florence, Italy.
  30. Edoardo Maria Ponti, Julia Kreutzer, Ivan Vulić, and Siva Reddy. 2021. Modelling latent translations for cross-lingual transfer. In Arxiv. find
  31. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In ACL.
  32. Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In ACL, pages 151–164, Florence, Italy. Association for Computational Linguistics.
  33. Machel Reid and Mikel Artetxe. 2021. PAR-ADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining. arXiv preprint arXiv:2108.01887. — [arXiv:2108.01887](https://arxiv.org/abs/2108.01887) find
  34. Shruti Rijhwani, Antonios Anastasopoulos, and Graham Neubig. 2020. OCR Post Correction for Endangered Language Texts. In EMNLP, Online. Association for Computational Linguistics. find
  35. Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation. In Proceedings of EMNLP 2021.
  36. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569–631. find
  37. Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low-resource languages. In EMNLP-Findings, Online. Association for Computational Linguistics. find
  38. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In EMNLP. *Figure 6: Improvements of using combined lexicons compared to PanLex lexicons for Pseudo MLM. Languages with fewer PanLex lexicons tend to benefit more from the combined lexicons.* A *…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2022-expanding-pretrained-models-thousands-more/