Elamur Research⌕
☰
home / papers / 2020 / Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinya

Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinya

Summary

Building a transformer from scratch for every low-resource language costs too much, and multilingual models such as mBERT cover only about a hundred languages, with weaker results for the poorly represented ones. The authors ask whether a model pretrained only on English can be adapted to a new language by replacing just its lexical layer. Their recipe has three steps. First, learn a SentencePiece vocabulary from unlabelled target-language text. Second, train context-free Word2Vec vectors for that vocabulary. Third, fine-tune the English transformer (BERT, XLNet or mBERT) on labelled target data while the new embeddings stay frozen. They test this on a new Tigrinya sentiment dataset, written in Ge'ez script and taken from YouTube comments, and on the CLS benchmark. English XLNet transferred this way is competitive with mBERT and with TigXLNet, a model pretrained only on Tigrinya. The work suggests that a strong monolingual encoder can serve a language with no shared vocabulary, and it releases the first public Tigrinya sentiment resource.

Key points

  • With only 10k Tigrinya training examples, transferred English XLNet reached an F1 of 78.88%. That is roughly 10 points above BERT and 7 above mBERT under the same setup.
  • The new Tigrinya dataset was built in two parts:
    • Training set: about 30k automatically labelled comments from Eritrean and Ethiopian music and short-film channels, expanded to about 50k through augmentation.
    • Test set: 4k items, balanced 2k positive and 2k negative, kept only where two professional annotators agreed.
  • Augmentation follows the EDA operations: synonym swap, random swap, insertion and deletion. Synonyms are chosen by Word2Vec similarity instead of a thesaurus.
  • On CLS (Amazon reviews in German, French and Japanese across the Music, Books and DVD domains), English XLNet came close to mBERT and beat it on one Japanese subset.
  • Tigrinya is a Semitic SOV language with root-and-pattern morphology and around 275 Fidel symbols. The authors tokenize the raw Ge'ez script with SentencePiece rather than transliterating it.

Interesting ideas

  • I1. Swap only the lexical layer: keep a pretrained encoder's upper layers and plug in freshly learned subword embeddings for a new vocabulary, which avoids full pretraining. — For Mova (tree): gparse could be adapted to a new domain or a non-standard English variety by retraining only its token embeddings, leaving the parser body untouched.
  • I2. Freeze the injected embeddings during fine-tuning so that the small labelled set adapts the encoder to them, instead of distorting vectors learned on a large corpus. — For Mova (learn): when training on small silver sets, freezing pretrained lexical vectors could reduce overfitting and keep behaviour easier to trace.
  • I3. Build a cheap, large training set with automatic labels, but keep a small test set that two annotators labelled independently and agreed on. — For Mova (learn): this matches the silver-versus-gold split; evaluation sets could be restricted to items where independent judges (human or LLM tiers) agree.
  • I4. Augment text by replacing words with their nearest neighbours in an embedding space when no thesaurus or MT system is available. — For Mova (learn): such perturbed sentences could act as robustness probes or negative controls, checking whether a frame or answer survives near-synonym swaps.
  • I5. Benchmark the transferred model against one pretrained natively on the target language, and against a multilingual model that saw that language, to bracket how good transfer can be. — For Mova (answer): tests of the school-course or fable readers could include similar reference points, so that gains are judged against both a native upper bound and a generic baseline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

9 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2006.07698 · arXiv 2006.07698

@misc{woubie2020transferring,
  title = {{Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinya}},
  author = {Woubie;, Abrhalei Tela; Abraham and Hautamaki, Ville},
  year = {2020},
  doi = {10.48550/arxiv.2006.07698},
  eprint = {2006.07698},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.07698}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Bird, E. Klein, and E. Loper, Natural Language Processing with Python, 1st ed. OReilly Media, Inc., 2009. find
  2. Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean, “Google’s neural machine translation system: Bridging the gap between human and machine translation,” CoRR, vol. abs/1609.08144, 2016. [Online]. Available: http://arxiv.org/abs/1609.08144 ↗
  3. B. Liu, Sentiment Analysis and Opinion Mining. Morgan & Claypool Publishers, 2012. find
  4. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,” Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016. [Online]. Available: http://dx.doi.org/10.18653/v1/D16-1264 ↗ our card
  5. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc., 2017, pp. 5998–6008. [Online]. Available: http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf ↗
  6. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://www.aclweb.org/anthology/N19-1423 ↗
  7. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” in Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 5753–5763. [Online]. Available: https://arxiv.org/pdf/1906.08237.pdf ↗ our card
  8. S. Ruder, A. Søgaard, and I. Vulić, “Unsupervised cross-lingual representation learning,” in Proceedings of ACL 2019, Tutorial Abstracts, 2019, pp. 31–38. find
  9. C. Wang, M. Li, and A. J. Smola, “Language models with transformers,” CoRR, vol. abs/1904.09408, 2019. [Online]. Available: http://arxiv.org/abs/1904.09408 ↗
  10. A. Conneau and G. Lample, “Cross-lingual language model pretraining,” in Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 7059–7069. [Online]. Available: http://papers.nips.cc/paper/8928-cross-lingual-language-model-pretraining.pdf ↗
  11. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmn, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” 2019. find
  12. W. de Vries, A. van Cranenburgh, A. Bisazza, T. Caselli, G. van Noord, and M. Nissim, “Bertje: A dutch bert model,” 2019. find our card
  13. A. Virtanen, J. Kanerva, R. Ilo, J. Luoma, J. Luotolahti, T. Salakoski, F. Ginter, and S. Pyysalo, “Multilingual is not enough: Bert for finnish,” 2019. find our card
  14. K. K, Z. Wang, S. Mayhew, and D. Roth, “Cross-lingual ability of multilingual bert: An empirical study,” 2019. find
  15. M. Artetxe, S. Ruder, and D. Yogatama, “On the cross-lingual transferability of monolingual representations,” 2019. find our card
  16. Y. Tedla and K. Yamamoto, “Morphological segmentation with lstm neural networks for tigrinya,” 2018. find
  17. R. Hetzron, The Semitic Languages. 270 Madison Ave, New York USA 10016: Routledge, 1997, pp. 426–430. find
  18. O. Osman and Y. Mikami, “Stemming Tigrinya words for information retrieval,” in Proceedings of COLING 2012: Demonstration Papers. Mumbai, India: The COLING 2012 Organizing Committee, Dec. 2012, pp. 345–352. [Online]. Available: https://www.aclweb.org/anthology/C12-3043 ↗
  19. M. Tadesse and Y. Assabie, “Trilingual sentiment analysis on social media,” Master’s thesis, Univeristy of Addis Ababa, 2018. [Online]. Available: http://etd.aau.edu.et/handle/ 123456789/17926 ↗
  20. Y. K. Tedla, K. Yamamoto, and A. Marasinghe, “Tigrinya part-of-speech tagging with morphological patterns and the new nagaoka tigrinya corpus,” International Journal of Computer Applications, vol. 146, no. 14, pp. 33–41, Jul 2016. [Online]. Available: http://www.ijcaonline.org/archives/volume146/ number14/25468-2016910943 ↗
  21. A. Sahle, A comprehensive Tigrinya grammar. Asmara, Eritrea: Lawrenceville NJ: Red Sea Press, Inc., 1998, pp. 71–72. find
  22. T. Kudo and J. Richardson, “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” 2018. find our card
  23. Z. Chi, L. Dong, F. Wei, X.-L. Mao, and H. Huang, “Can monolingual pretrained models help cross-lingual classification?” 2019. find
  24. T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” 2013. find
  25. P. Prettenhofer and B. Stein, “Cross-language text classification using structural correspondence learning,” in Proceedings of the ACL, 2010. find
  26. A. Sugiyama and N. Yoshinaga, “Data augmentation using back-translation for context-aware neural machine translation,” in Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019). Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 35–44. [Online]. Available: https://www.aclweb.org/anthology/D19-6504 ↗
  27. J. Wei and K. Zou, “Eda: Easy data augmentation techniques for boosting performance on text classification tasks,” in EMNLP/IJCNLP, 2019. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tela-2020-transferring-monolingual-model-low-resource/