Elamur Research⌕
☰
home / papers / 2020 / Can Multilingual Language Models Transfer to an Unseen Dialect? A Case Study on North African Arabizi

Can Multilingual Language Models Transfer to an Unseen Dialect? A Case Study on North African Arabizi

Summary

The paper asks whether a multilingual pretrained encoder can handle a language variety it never saw during pretraining. The test case is Narabizi, the Algerian Arabic dialect as written on social media. It uses Latin script, has no spelling norm and mixes heavily with French. The authors fine-tune mBERT as a POS tagger (softmax head) and as a dependency parser (biaffine head) on 1,200 sentences from one source language: Maltese, French, English, MSA or Vietnamese. They then apply each model directly to the Narabizi UD treebank. They also test a variant that first continues masked-language-model training on about 49.5k raw Narabizi sentences.

The zero-shot models clearly beat trivial bottom lines. Maltese, itself absent from pretraining, gives the best zero-shot scores. Raw-text adaptation raises results for almost every source, and it makes transfer from MSA across scripts workable. Control models show that mBERT's multilingual pretraining is what drives the transfer. This matters because a usable NLP pipeline can be started for vernacular varieties that have almost no annotated data.

Key points

  • Data. The Narabizi treebank (Seddah et al., 2020) has 1,434 UD-annotated sentences, split 1172/146/178, with word-level language IDs. The raw corpus has 49,546 sentences. Training tokens are roughly 65% Narabizi and 34% French.
  • Baselines.
    • A supervised StanfordNLP model reaches 84.20 POS accuracy and 52.84 UAS.
    • Bottom lines are 20.49 POS (majority class) and 18.71 UAS (attach each word to its left neighbor).
    • Only UAS is reported, because label conventions differ between treebanks, so LAS would not be comparable.
  • Zero-shot mBERT+TASK.
    • Maltese gives 35.13 POS / 40.04 UAS.
    • French gives 33.12 / 38.54.
    • Supervised training on Narabizi itself reaches 81.60 / 66.84.
  • Effect of mBERT+MLM+TASK.
    • On average it adds +10.15 POS and +4.10 UAS across sources.
    • With French as source, POS rises to 47.32.
    • With MSA as source, UAS rises from 28.23 to 34.34, so cross-script transfer starts working.
  • Controls. For both French and Maltese sources, mBERT beats RoBERTa, CamemBERT and a randomly initialized Transformer. The gap is largest for Maltese.

Interesting ideas

  • I1. The authors split the test data into buckets by the share of native-dialect tokens per sentence. This exposes that French-trained models succeed mostly on the French-heavy part, which an average score would hide. — For Mova (learn): Evaluations could be bucketed by a measurable trait of the input, such as the rate of informal or out-of-lexicon tokens. That would show which kinds of sentences actually drive a score gain.
  • I2. Continuing masked-LM training on a modest amount of raw target text, before any task fine-tuning, cheaply narrows the lexical gap between training and test domains. — For Mova (tree): gparse could be adapted on raw text from its target genres, such as fables or school-course texts, before parser training, with no extra annotation.
  • I3. When label schemes diverge across UD treebanks, the paper reports only unlabeled attachment rather than mixing incompatible labels. — For Mova (tree): When comparing parsers across treebanks or across UD dialect conversions, UAS could be reported separately from LAS. This keeps convention mismatches from looking like parsing errors.
  • I4. To attribute transfer to multilingual pretraining, the authors compare against monolingual models and a randomly initialized model with the same architecture. — For Mova (learn): Every claimed improvement from a teacher tier or silver-data source could be paired with a random-init or no-pretraining control as a negative check.
  • I5. Trivial bottom lines, such as the majority tag and attach-to-left-neighbor, define the level any claim of real transfer must exceed. — For Mova (learn): Evaluation reports could always include these trivial predictors. A low score could then be told apart from no learning at all.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Universal Dependencies · Treebanks

Cited by

12 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2005.00318 · arXiv 2005.00318

@misc{muller2020can,
  title = {{Can Multilingual Language Models Transfer to an Unseen Dialect? A Case Study on North African Arabizi}},
  author = {Muller, Benjamin and Sagot, Benoit and Seddah, Djame},
  year = {2020},
  doi = {10.48550/arxiv.2005.00318},
  eprint = {2005.00318},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.00318}
}

References

31 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Djegdjiga Amazouz, Martine Adda-Decker, and Lori Lamel. 2019. Addressing code-switching in french/algerian arabic speech. In Interspeech 2017, pages 62–66. find
  2. Kelsey Ball and Dan Garrette. 2018. Part-of-speech tagging for code-switched, transliterated texts without explicit language identification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3084–3089, Brussels, Belgium. Association for Computational Linguistics. find
  3. Slavomı́r Čéplö, Ján Bátora, Adam Benkato, Jiřı́ Milička, Christophe Pereira, and Petr Zemánek. 2016. Mutual intelligibility of spoken maltese, libyan arabic, and tunisian arabic functionally tested: A pilot study. Folia Linguistica, 50(2):583–628. find
  4. Özlem Çetinoğlu and Çağrı Çöltekin. 2016. Part of speech annotation of a Turkish-German code-switching corpus. In Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016), pages 120–130, Berlin, Germany. Association for Computational Linguistics. find
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  6. Ryan Cotterell, Adithya Renduchintala, Naomi Saphra, and Chris Callison-Burch. 2014. An algerian arabic-french code-switched corpus. In Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools Workshop Programme. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Timothy Dozat and Christopher D Manning. 2016. Deep biaffine attention for neural dependency parsing. arXiv preprint arXiv:1611.01734. ↗ our card
  9. Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kardas, Sylvain Gugger, and Jeremy Howard. 2019. Multifit: Efficient multi-lingual language model fine-tuning. arXiv preprint arXiv:1909.04761. ↗
  10. Mona Farrag. 2012. Arabizi: a writing variety worth learning? an exploratory study of the views of foreign learners of arabic on arabizi. Master’s thesis, School of Humanities and Social Sciences, American University in Cairo, Cairo, Egypt. find
  11. Hila Gonen and Yoav Goldberg. 2018. Language modeling for code-switching: Evaluation, integration of monolingual data, and discriminative training. arXiv preprint arXiv:1810.11895. ↗
  12. Nizar Habash. 2010. Introduction to Arabic Natural Language Processing. Morgan and Claypool. find
  13. Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised domain adaptation of contextualized embeddings: A case study in early modern english. arXiv preprint arXiv:1904.02817. ↗
  14. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. ↗
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  16. Teresa Lynn and Kevin Scannell. 2019. Code-switching in irish tweets: A preliminary analysis. In Proceedings of the Celtic Language Technology Workshop, pages 32–40, Dublin, Ireland. European Association for Machine Translation. find
  17. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoı̂t Sagot. 2019. CamemBERT: a Tasty French Language Model. arXiv e-prints, page arXiv:1911.03894. ↗ our card
  18. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 92–97, Sofia, Bulgaria. Association for Computational Linguistics. find
  19. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). ↗ our card
  20. Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1659–1666. find our card
  21. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? arXiv preprint arXiv:1906.01502. ↗ our card
  22. Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D Manning. 2019. Universal dependency parsing from scratch. arXiv preprint arXiv:1901.10457. ↗
  23. Sayahi. 2014. The languages of the Maghreb. In Diglossia and Language Contact: Language Variation and Change in North Africa. Cambridge: Cambridge University Press. find
  24. Djamé Seddah, Farah Essaidi, Amal Fethi, Matthieu Futeral, Benjamin Muller, Pedro Javier Ortiz Suárez, Benoı̂t Sagot, and Abhishek Srivastava. 2020. Building a user-generated content north-african arabizi treebank: Tackling hell. In 59th Annual Meeting of the Association for Computational Linguistics (ACL), Seattle, USA. find
  25. Thamar Solorio and Yang Liu. 2008. Part-of-Speech tagging for English-Spanish code-switched text. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 1051–1060, Honolulu, Hawaii. Association for Computational Linguistics. find
  26. Clara Vania, Yova Kementchedjhieva, Anders Søgaard, and Adam Lopez. 2019. A systematic comparison of methods for low-resource dependency parsing on genuinely low-resource languages. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1105–1116, Hong Kong, China. Association for Computational Linguistics. find
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  28. Yogarshi Vyas, Spandana Gella, Jatin Sharma, Kalika Bali, and Monojit Choudhury. 2014. POS tagging of English-Hindi code-mixed social media content. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 974–979, Doha, Qatar. Association for Computational Linguistics. find our card
  29. Zihan Wang, Stephen Mayhew, Dan Roth, et al. 2019. Cross-lingual ability of multilingual BERT: An empirical study. arXiv preprint arXiv:1912.07840. ↗
  30. Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Emerging cross-lingual structure in pretrained language models. arXiv preprint arXiv:1911.01464. ↗ our card
  31. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong, China. Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/muller-2020-multilingual-language-models-transfer-unseen/