Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.mrl-1.11.pdf
Summary
Large multilingual models such as mBERT and XLM-R need huge corpora. African languages make up only a sliver of that data, and it is widely assumed that low-resource languages need higher-resource ones to learn from. The authors test this assumption with AfriBERTa, a transformer trained from scratch only on African languages: 11 in total, counting the code-mixed Gahuza as Kinyarwanda plus Kirundi. They built a compact corpus from BBC News and Common Crawl, then deduplicated it, filtered short lines and removed wrong-language text by hand. They pretrained small, base and large variants with masked language modelling and a SentencePiece tokenizer. On text classification, AfriBERTa beats mBERT and XLM-R by a wide margin. On NER, it wins for several languages and stays close elsewhere. The baselines remain strong on Nigerian Pidgin because of its closeness to English. The study shows that compact models trained on clean data from related languages can be a practical, cheaper path for under-served languages.
Key points
- The pretraining corpus is about 0.94 GB, with roughly 108.8M tokens and 5.45M sentences. By comparison, XLM-R used about 2395 GB (164.0B tokens) and mBERT about 100 GB (12.8B tokens).
- AfriBERTa base has 112M parameters, compared with 270M for XLM-R base and 172M for mBERT.
- For text classification, AfriBERTa large leads by more than 10 F1 on Yorùbá and up to 7 F1 on Hausa, despite having about half of XLM-R's Hausa data.
- Training uses masked language modelling without next-sentence prediction, with language-alternating batches and a SentencePiece unigram tokenizer sampled with α = 0.3.
- For 4 of the covered languages, this work offers the first language models ever built. The authors have made the code, data and models publicly available at github.com/keleog/afriberta.
Cited by
- Do Diacritics Matter? Evaluating the Impact of Arabic Diacritics on Tokenization and LLM Benchmarks
- SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects
- Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
- Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya
- Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
Cite
@inproceedings{ogueji2021small,
title = {{Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages}},
author = {Ogueji, Kelechi and Zhu, Yuxin and Lin, Jimmy},
booktitle = {Proceedings of the 1st Workshop on Multilingual Representation Learning},
year = {2021},
doi = {10.18653/v1/2021.mrl-1.11},
url = {https://doi.org/10.18653/v1/2021.mrl-1.11}
}References
- David Ifeoluwa Adelani, Jade Z. Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Oga… find our card