Elamur Research⌕
☰
home / papers / 2026 / MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages

MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages

Summary

South Africa has eleven official written languages, nine of them low-resource Bantu languages. The authors know of no public decoder-only model that targets all eleven. They also ask whether instruction-style adaptation still helps when the model is small. To address both, they built MzansiText, a multilingual pretraining corpus. It merges several public sources and cleans them with a deterministic, reproducible Datatrove pipeline: language identification, normalisation, deduplication, safety screening and quality filtering. On this corpus they trained MzansiLM from scratch, a 125M-parameter LLaMA-style decoder using the MobileLLM configuration. Next they tested three ways of adapting the model: tuning it for one task in a single language, tuning it for one task across multiple languages, and broad instruction tuning spanning many tasks. Task-specific supervision proved most effective at this size. On some generation tasks it rivals models more than ten times larger. Encoder-based models still lead on classification and structured prediction, and few-shot reasoning stays near chance. The release gives South African NLP an open, reproducible decoder-only baseline and practical advice on adapting small models.

Key points

  • Corpus sources: MzansiText draws on mC4, CulturaX, WURA, Glot500-c, NCHLT Text, CC100, ParaCrawl and Inkuba-Mono.
    • Deduplication removes 4.0% of words, and filtering then removes 22.9% of the remaining text.
  • Language balance: about 84% of tokens are Afrikaans and English. A more balanced mixture was tested but degraded downstream results, so the natural distribution was kept.
  • Model: MzansiLM has 125M parameters, a 2,048-token context and a 65,536-token tokenizer.
  • Main results: monolingual finetuning reaches 20.65 BLEU on isiXhosa data-to-text generation. Multilingual finetuning reaches 78.5% macro-F1 on isiXhosa news classification.
    • Downstream evaluation covers eight of the eleven languages.
  • Releases: the corpus (anrilombard/mzansi-text) and model (anrilombard/mzansilm-125m) are on HuggingFace, and the code is at github.com/Anri-Lombard/sallm.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4g7i3kqszquh

@inproceedings{lombard2026mzansitext,
  title = {{MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages}},
  author = {Lombard, Anri M. and Mawere, Simbarashe and Aina, Temi and Wolff, Ethan and Gumede, Sbonelo and Novick, Elan and Meyer, Francois and Buys, Jan},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4g7i3kqszquh},
  url = {https://doi.org/10.63317/4g7i3kqszquh}
}

References

51 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen You… find
  2. David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee. 2024. Sib-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Proceedings of the 18th Conference of the Eu… find our card
  3. David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo, Doreen Nixdorf, Chris Chinenye Emezue, Sana Al-azzawi, Blessing Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi, Tu… find
  4. David Ifeoluwa Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba O. Alabi, Shamsuddeen H. Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy B… find
  5. David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba Oluwadara Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Ijeoma Chukwuneke, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson Koffi Kalipe, Jonathan Mukiibi, Salomon Kab… find
  6. Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. Adapting pre-trained language models to african languages via multilingual adaptive fine-tuning. In Proceedings of the AfricaNLP Workshop at ICLR. find our card
  7. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. 2024. Smollm - blazingly fast and remarkably powerful. find
  8. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual M… find our card
  9. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winte… find
  10. Happy Buzaaba, Alexander Wettig, David Ifeoluwa Adelani, and Christiane Fellbaum. 2025. find
  11. Lugha-llama: Adapting large language models for african languages. CoRR, abs/2504.06536. DBLP entry for the arXiv version. — [arXiv:2504.06536](https://arxiv.org/abs/2504.06536) find
  12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT. find
  14. Cheikh M. Bamba Dione, David Ifeoluwa Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue, Perez Ogayo, Anuoluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jonathan Mukiibi, Blessing Sibanda, Bonaventure F. P. Dossou, Andiswa Buku… find
  15. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… find
  16. Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kadras, Sylvain Gugger, and Jeremy Howard. 2019. MultiFiT: Efficient multi-lingual language model fine-tuning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferenc… find
  17. EleutherAI. 2024. lm-evaluation-harness (v0.4.3). find
  18. Hugging Face DataTrove Team. 2024. Datatrove: Large-scale data processing for llm training. Open-source library for processing, filtering, and deduplicating web-scale text. find
  19. Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. 2024. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. find
  20. Francois Meyer and Jan Buys. 2024. Triples-to-isixhosa (t2x): Addressing the challenges of low-resource agglutinative data-to-text generation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16841–168… find
  21. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the Joint International Conference on Computat… find our card
  22. Derek Nurse and Gérard Philippson, editors. 2004. The Bantu Languages. Routledge. Overview of Bantu typology, including noun-class systems. find
  23. Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 116–126, Punta Cana, Dominican Republic. Associati… find our card
  24. Odunayo Ogundepo, Tajuddeen R. Gwadabe, Clara E. Rivera, Jonathan H. Clark, Sebastian Ruder, David Ifeoluwa Adelani, Bonaventure F. P. Dossou, Abdou Aziz Diop, Claytone Sikasote, Gilles Hacheme, Happy Buzaaba, Ignatius Ezeani, Rooweither Mabuya, Salomey Osei, Chris Emezue, Albert Njoroge Kahira, Sha… find
  25. Odunayo Ogundepo, Akintunde Oladipo, Mofetoluwa Adeyemi, Kelechi Ogueji, and Jimmy Lin. 2022. Afriteva: Extending “small data” pretraining approaches to sequence-to-sequence models. In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pages 126–135, Dub… find
  26. Toyib Ogunremi, Serah Akojenu, Anthony Soronnadi, Olubayo Adekanmbi, and David Ifeoluwa Adelani. 2024. Afrihg: News headline generation for african languages. OpenReview. find
  27. Akintunde Oladipo, Mofetoluwa Adeyemi, Orevaoghene Ahia, Abraham Toluwalase Owodunni, Odunayo Ogundepo, David Ifeoluwa Adelani, and Jimmy Lin. 2023. Better quality pre-training data and t5 models for african languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pr… find
  28. Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557. find our card
  29. Jack W. Rae et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. find
  30. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. find
  31. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. CoRR, abs/1511.06709. find
  32. Statistics South Africa. 2024. find
  33. Census 2022 in brief. Technical report, Stats SA. Table 3.8 & Figure 3.9. find
  34. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2021. Multilingual translation from denoising pre-training. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3450–3466, Online. Association for Computational… find our card
  35. Atnafu Lambebo Tonja, Bonaventure F. P. Dossou, Jessica Ojo, Jenalea Rajab, Fadel Thior, Eric Peter Wairagala, Aremu Anuoluwapo, Pelonomi Moiloa, Jade Abbott, Vukosi Marivate, and Benjamin Rosman. 2024a. InkubaLM: A small language model for low-resource african languages. arXiv preprint arXiv:2408.1… — [arXiv:2408.17024](https://arxiv.org/abs/2408.17024) find
  36. Atnafu Lambebo Tonja, Bonaventure F. P. Dossou, Jessica Ojo, Jenalea Rajab, Fadel Thior, Eric Peter Wairagala, Anuoluwapo Aremu, Pelonomi Moiloa, Jade Abbott, Vukosi Marivate, and Benjamin Rosman. 2024b. Inkubalm: A small language model for low-resource african languages. arXiv. find
  37. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv pr… find
  38. Kosei Uemura, Mahe Chen, Alex Pejovic, Chika Maduabuchi, Yifei Sun, and En-Shiun Annie Lee. 2024. AfriInstruct: Instruction tuning of african languages for diverse tasks. In Findings of EMNLP, pages 13571–13585. find
  39. Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya Model: An instruction finetune… find our card
  40. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of ACL, pages 13484– 13508. find
  41. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In Proceedings of the Tenth International Conference on Learning Representations (ICLR), Virtual Conference. find
  42. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Édouard Grave. 2020. Ccnet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC 2020), pages 4003–… find our card
  43. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… find
  44. Hao Yu, Jesujoba Oluwadara Alabi, Andiswa Bukula, Jian Yun Zhuang, En-Shiun Annie Lee, Tadesse Kebede Guge, Israel Abebe Azime, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson Koffi Kalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither M… find
  45. Language Resource References Eiselen, Roald and Puttkammer, Martin. 2014. SADiLaR NCHLT Text Corpora. SADiLaR Language Resource Management Agency. SADi-LaR. PID https://hdl.handle.net/20.500.12185/1. ↗
  46. Imani, Ayyoob and Lin, Peiqin and Kargaran, Amir Hossein and Severini, Silvia and Jalili Sabet, Masoud and Kassner, Nora and Ma, Chunlan and Schmid, Helmut and Martins, André and Yvon, François and Schütze, Hinrich. 2023. Glot500-c. CIS, LMU Munich. Hugging Face. PID https://huggingface.co/datasets/… ↗
  47. Jacaranda Health and Stanslaus, Mwongela and Jay, Patel and Rajasekharan, Lusiji Sathy and Lyvia and Francesco, Piccino and Mfoniso, Ukwak and Ellen, Sebastian. 2024. AfroLlama V1: An instruction-tuned Llama model for African languages. Hugging Face. PID https://huggingface.co/Jacaranda/AfroLlama_V1… ↗
  48. Lelapa AI - Fundamental Research Team. 2024. Inkuba-Mono: Monolingual Corpora for African Languages. Lelapa AI. Hugging Face. PID https://huggingface.co/datasets/lelapa/Inkuba-Mono. Dataset accompanying InkubaLM. ↗
  49. Nguyen, Thuat and Nguyen, Chien Van and Lai, Viet Dac and Man, Hieu and Ngo, Nghia Trung and Dernoncourt, Franck and Rossi, Ryan A. and Nguyen, Thien Huu. 2024. CulturaX. UONLP. Hugging Face. PID https://huggingface.co/datasets/uonlp/CulturaX. ↗
  50. Oladipo, Akintunde and Adeyemi, Mofetoluwa and Ahia, Orevaoghene and Owodunni, Abraham Toluwalase and Ogundepo, Odunayo and Adelani, David Ifeoluwa and Lin, Jimmy. 2023. WURA. Castorini. Hugging Face. PID https://huggingface.co/datasets/castorini/wura. ↗
  51. ParaCrawl Project. 2020. ParaCrawl Release v7.1. ParaCrawl Project. ParaCrawl. PID https://paracrawl.eu/v7-1. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lombard-2026-mzansitext-mzansilm-open-corpus-decoder/