Elamur Research⌕
☰
home / papers / 2026 / Sample-Size Scaling of the African Languages NLI Evaluation

Sample-Size Scaling of the African Languages NLI Evaluation

Paper · 2026 · Anuj Tiwari, Oluwapelumi Ogunremu, Terry Oko-odion et al.

Summary

Labelled data for African languages is scarce, and it is unclear whether more annotated examples reliably yield better or more trustworthy NLI results. The authors study this with the AfriXNLI benchmark across 16 African languages. They compare two multilingual encoders of about 0.6B parameters each: XLM-R Large fine-tuned on XNLI and AfroXLM-R Large. The abstract describes the study as a training-data scaling study, but the experimental sections show something different. The models are never retrained. Instead, the authors randomly subsample the AfriXNLI test splits at sizes from 50 to 500 examples and repeat this several times. They then report the mean and standard deviation of accuracy, precision and F1. The findings concern how stable evaluation is, not how much models learn. Scaling turns out to be strongly language-dependent and often non-monotonic. Small subsets tend to overstate accuracy, and variance stays high below roughly 300 examples. The paper therefore argues that African NLI benchmarks should report variance across subsamples and use sufficiently large, language-aware test sets.

Key points

  • Setup: The study covers 16 AfriXNLI languages and two models, XLM-R Large (XNLI-fine-tuned) and AfroXLM-R Large. Test subsets range from 50 to 500 examples, with repeated random draws at each size.
  • Variance: Accuracy variance is highest at 50–100 samples. It levels off at about 300 samples for both models.
  • Language-specific trends:
    • Yoruba starts with high accuracy at 50 examples and declines steadily as more samples are added.
    • Kinyarwanda improves slightly up to about 150 examples, then flattens.
    • Both patterns appear under both models.
  • Saturation points: These mark where mean accuracy stays within ±0.5%. They range from about 200–250 samples for Shona, Xhosa and Yoruba under XLM-R up to 450 for several languages. Wolof never stabilises within 500 samples under XLM-R.
  • Recommendations:
    • Report mean ± SD over multiple subsamples.
    • Add per-class metrics.
    • Use at least 300 evaluation samples, and treat smaller sets as high-noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Noida Institute of Engineering and TechnologyGreater Noida · India2 works

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Anuj Tiwari, Oluwapelumi Ogunremu, Terry Oko-odion, Jesujuwon Egbewale and Hannah Sopuruchi Nwokocha. 2026. Sample-Size Scaling of the African Languages NLI Evaluation. Proceedings of the 7th Workshop on African Natural Language Processing (AfricaNLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.africanlp-main.22

@inproceedings{tiwari2026sample,
  title = {{Sample-Size Scaling of the African Languages NLI Evaluation}},
  author = {Anuj Tiwari, Oluwapelumi Ogunremu, Terry Oko-odion, Jesujuwon Egbewale and Hannah Sopuruchi Nwokocha},
  booktitle = {Proceedings of the 7th Workshop on African Natural Language Processing (AfricaNLP 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.africanlp-main.22},
  url = {https://doi.org/10.18653/v1/2026.africanlp-main.22}
}

References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ife Adebara and Muhammad Abdul-Mageed. 2022. Towards afrocentric NLP for African languages: Where we are and where we can go. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3814–3841. Association for Computational Linguistics… find
  2. Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Inciarte. 2022. AfroLID: A neural language identification tool for African languages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1958–1981. Association for Computational Linguis… find
  3. David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, and 1 others. 2021. MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics, pages 1116–1131. 29th International Conference on Computational Linguistics. International Committee on C…
  4. Masakhane NLP Community. 2024. Afrixnli: Dataset. find
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, and 1 others. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451. find
  6. Joe Davison. 2020. Xlm-roberta large fine-tuned on xnli. Hugging Face model. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  8. Roald Eiselen and Tanja Gaustad. 2023. Deep learning and low-resource languages: How much data is enough? a case study of three linguistically distinct south african languages. In Proceedings of the Fourth Workshop on Resources for African Indigenous Languages (RAIL 2023), pages 42–53. Association f… find
  9. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, and 1 others. 2022. Training compute-optimal large language models. In Proceedings of NeurIPS.
  10. Kedir Yassin Hussen, Walelign Tewabe Sewunetie, Abinew Ali Ayele, Sukairaj Hafiz Imam, Shamsuddeen Hassan Muhammad, and Seid Muhie Yimam. 2025. The state of large language models for african languages: Progress and challenges. ArXiv, abs/2506.02280. — [arXiv:2506.02280](https://arxiv.org/abs/2506.02280) find
  11. Jared Kaplan, Sam McCandlish, T. J. Henighan, and 1 others. 2020. Scaling laws for neural language models. ArXiv, abs/2001.08361.
  12. Niklas Muennighoff, Sébastien Candido, Jonathan Va-malké, and 1 others. 2023. Scaling data-constrained language models. https://sl1nk.com/wyS64. ↗
  13. David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, and 1 others. 2025. IrokoBench: A new benchmark for African languages in the age of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Lan…
  14. Jessica Ojo, Odunayo Ogundepo, Akintunde Oladipo, and 1 others. 2025. AfroBench: How good are large language models on African languages? In Findings of the Association for Computational Linguistics: ACL 2025, pages 19048–19095. Association for Computational Linguistics. find
  15. A Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and 1 others. 2022. Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning". In Proceedings of the

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tiwari-2026-sample-size-scaling-african-languages/