Elamur Research
☰
home / papers / 2022 / Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning

Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.coling-1.382.pdf

Summary

Multilingual PLMs such as XLM-R still perform poorly on many African languages, especially those absent from pre-training. The standard fix, language adaptive fine-tuning (LAFT), produces one specialized model per language. That approach uses a lot of storage and weakens cross-lingual transfer. The authors instead propose multilingual adaptive fine-tuning (MAFT): they continue masked-language-model training of one model on 17 African languages plus English, French and Arabic at the same time. They also prune embedding-layer vocabulary for scripts other than Latin and Ge'ez, which roughly halves the size of XLM-R.

They evaluated the method with AfriBERTa and XLM-R on NER, news topic classification and sentiment classification. The single adapted model performs on par with per-language LAFT while needing far less disk space. It also gives a stronger base for parameter-efficient methods such as MAD-X 2.0 and LT-SFT. The work also introduces a new news-topic dataset, ANTC. Together, these results provide compact, reusable models for African NLP, which matters for researchers limited to modest hardware like free Google Colab.

Key points

  • MAFT covers 20 languages: 17 African languages (from Afrikaans and Amharic to Yorùbá and isiZulu) and three widely used high-resource languages (English, French, Arabic). The text comes from the mT5/Common Crawl corpus, BBC, VOA and other African news sites.
  • Pruning the vocabulary to Latin and Ge'ez scripts cuts model parameters by about 50% without losing competitiveness against LAFT.
  • ANTC (African News Topic Classification) is a new dataset for Lingala, Naija, Malagasy, Somali and isiZulu. It is built from VOA, BBC, Global Voices and Isolezwe categories.
  • In zero-shot NER transfer from CoNLL03 to MasakhaNER, training language adapters/SFTs on news text instead of Wikipedia improves scores by +4.2 (MAD-X 2.0) and +3.7 (LT-SFT).
  • The best results come from LT-SFT on AfroXLMR-base, which sometimes beats full supervision (e.g., kin, lug, wol, yor). AfroXLMR-base/small/mini, the adapters and the SFTs are released on HuggingFace (Davlan), with code at uds-lsv/afro-maft.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

147 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 144 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alabi2022adapting,
  title = {{Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning}},
  author = {Alabi, Jesujoba O. and Adelani, David Ifeoluwa and Mosbach, Marius and Klakow, Dietrich},
  booktitle = {Proceedings of the 29th International Conference on Computational Linguistics},
  year = {2022},
  url = {https://aclanthology.org/2022.coling-1.382/}
}

References

44 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amine Abdaoui, Camille Pradel, and Grégoire Sigel. Load what you need: Smaller versions of mutililingual BERT. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pp. 119–123, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2… find
  2. David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andr… find
  3. David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Are… find our card
  4. Jesujoba Alabi, Kwabena Amponsah-Kaakyire, David Adelani, and Cristina España-Bonet. Massive vs. curated embeddings for low-resourced languages: the case of Yorùbá and Twi. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 2754–2762, Marseille, France, May 2020. Europea… find
  5. Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross-lingual transfer, 2021. find
  6. Israel Abebe Azime and Nebil Mohammed. An amharic news text classification dataset. abs/2103.05639, 2021. find
  7. ArXiv, Yassine Benajiba, Paolo Rosso, and José Miguel Benedı́Ruiz. Anersys: An arabic named entity recognition system based on maximum entropy. In Alexander Gelbukh (ed.), Computational Linguistics and Intelligent Text Processing, pp. 143–153, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg. I… find
  8. Michael Beukman. Analysing the effects of transfer learning on low-resourced named entity recognition performance. In 3rd Workshop on African Natural Language Processing, 2021. URL https://openreview.net/ forum?id=HKWMFqfN8b5. ↗
  9. Ethan C. Chau and Noah A. Smith. Specializing multilingual language models: An empirical study. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pp. 51–61, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.mrl-… find
  10. Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association… find
  11. Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. Rethinking embedding coupling in pre-trained language models. In International Conference on Learning Representations, 2021. URL https://openreview. net/forum?id=xpFFI\_NtgpW. find our card
  12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Com… find
  13. Davis David. Swahili : News classification dataset. Zenodo, December 2020. doi: 10.5281/zenodo. 5514203. URL https://doi.org/10.5281/ zenodo.5514203. The news version contains both train and test sets. — [doi:10.5281/zenodo](https://doi.org/10.5281/zenodo) find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… find
  15. Roald Eiselen. Government domain named entity recognition for South African languages. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pp. 3344–3348, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https://aclant… find
  16. ∀, Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa Berhe, Mofetoluw… find our card
  17. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https:// openreview.net/forum?id=rJl-b3RcF7. find
  18. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342–8360, Online, July 2020… find
  19. Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. ArXiv, abs/2111.09543, 2021. find our card
  20. Michael A. Hedderich, David Adelani, Dawei Zhu, Jesujoba Alabi, Udia Markus, and Dietrich Klakow. Transfer learning and distant supervision for multilingual transformer models: A study on African languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EM… find
  21. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Ma… find
  22. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4163–4174, Online, November 2020. Association for Computational… find
  23. Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. FastBERT: a self-distilling BERT with adaptive inference time. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6035– 6044, Online, July 2020a. Association for Computational Linguist… find
  24. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742, 2020b. doi: 10.1162/tacl a 00343. URL http… find
  25. Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, and Luo Si. VECO: Variable and flexible cross-lingual pre-training for language understanding and generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internat… find
  26. Processing (Volume 1: Long Papers), pp. 3980– 3994, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. acl-long.308. URL https://aclanthology. org/2021.acl-long.308. find
  27. Shamsuddeen Hassan Muhammad, David Ifeoluwa Adelani, Sebastian Ruder, Ibrahim Said Ahmad, Idris Abdulmumin, Bello Shehu Bello, Monojit Choudhury, Chris Chinenye Emezue, Saheed Abdullahi Salahudeen, Aremu Anuoluwapo, Alı́pio Jeorge, and Pavel Brazdil. Naijasenti: A nigerian twitter sentiment corpus f… find
  28. Benjamin Muller, Antonios Anastasopoulos, Benoı̂t Sagot, and Djamé Seddah. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisti… find our card
  29. Rubungo Andre Niyongabo, Qu Hong, Julia Kreutzer, and Li Huang. KINNEWS and KIRNEWS: Benchmarking cross-lingual text classification for Kinyarwanda and Kirundi. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 5507–5521, Barcelona, Spain (Online), December 2020.… find
  30. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference,… find
  31. Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pp. 116–126, Punta Cana, Dominican Republic, November 2021. As… find our card
  32. Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE-M: Enhanced multilingual representation by align-Chester Palen-Michel, June Kim, and Constantine Lignos. Multilingual open text 1.0: Public domain news in 44 languages. CoRR, abs/2201.05609, 2022. URL https://ar… — [arXiv:2201.05609](https://arxiv.org/abs/2201.05609) find
  33. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7654–7673, Online, November 2020. Association for Computat… find our card
  34. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. UNKs everywhere: Adapting multilingual language models to new scripts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10186– 10203, Online and Punta Cana, Dominican Republic, November 20… find
  35. Sara Rosenthal, Noura Farra, and Preslav Nakov. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pp. 502– 518, 2017. find our card
  36. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108, 2019. find
  37. Kathleen Siminyu, Godson Kalipe, Davor Orlic, Jade Z. Abbott, Vukosi Marivate, Sackey Freshia, Prateek Sibal, Bhanu Neupane, David Ifeoluwa Adelani, Amelia Taylor, Jamiil Toure Ali, Kevin Degila, Momboladji Balogoun, Thierno Ibrahima Diop, Davis David, Chayma Fourati, Hatem Haddad, and Malek Naski.… — [arXiv:2104.02516](https://arxiv.org/abs/2104.02516) find
  38. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2158–2170, Online, July 2020. Association for Computa… find our card
  39. Y. Tang, C. Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. Multilingual translation with extensible multilingual pretraining and finetuning. ArXiv, abs/2008.00401, 2020. find
  40. Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142–147, 2003. URL https://aclanthology.org/ W03-0419. ↗
  41. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volum… find
  42. Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. MiniLMv2: Multi-head self-attention relation distillation for compressing pre-trained transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2140–2151, Online, August 2021. Association for Comput… find
  43. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… find our card
  44. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015. find our card