Elamur Research⌕
☰
home / papers / 2026 / The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs

The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs

Paper · 2026 · Olaoye Anthony Somide

Summary

Commercial LLMs bill per token, generation time grows with output length, and context windows are measured in tokens. A text in one language can therefore cost more than the same text in another before any model quality matters. This paper measures that hidden penalty for African languages. It covers 20 languages from five families and three scripts (Latin, Ge'ez/Ethiopic, N'Ko) and runs 11 frontier and open tokenizers over parallel corpora, mainly FLORES-200+. Because every language carries the same content, any difference in token counts comes from the language and its script. For each language and tokenizer, fertility (tokens per word) is divided by the English fertility to give a premium. These premiums are then turned into money (USD and local currencies), latency multipliers and the usable share of the context window. Every African language costs more than English. Ethiopic and N'Ko are hit hardest, and no tokenizer removes the gap. This matters because the extra cost lands on markets that can least afford compute, and accuracy benchmarks alone do not reveal it.

Key points

  • Median premium over English is 1.88× on GPT-5 / o200k_base. N'Ko reaches 8.92×, and the Ethiopic and N'Ko scripts sit in the 7–9× range.
  • In deployment terms, N'Ko on GPT-5 costs up to 8.9× more and is slower by a similar factor (Amharic: 7.4×). The effective context can shrink to about 11% of what English gets.
  • Gemma 4 is the least costly tokenizer for these languages: it lowers the mean premium from 3.31× (cl100k_base) to 2.38×, but the premium never disappears.
  • Results barely change between corpora (FLORES vs SIB-200 Pearson r = 0.9998). Nigerian Pidgin is measured only on MAFAND-MT.
  • Released artifacts:
    • the afri-fertility tool (PyPI, Apache-2.0);
    • a results dataset on Hugging Face (CC-BY-4.0);
    • a leaderboard JSON;
    • a mitigation guide;
    • a cost model in NGN/ZAR/KES with three scenarios (bank assistant, clinical triage, SMS agricultural advice).

Interesting ideas

  • I1. The same counting rules are applied to every language (NFC normalization, UAX-29 word segmentation, special/BOS/EOS tokens excluded), and any fallback to a regex segmenter is written into the run metadata, so a hidden preprocessing change cannot skew comparisons.
  • I2. Running the same measurement on a second, independent corpus and checking that per-language results correlate almost perfectly shows the metric reflects the system rather than the particular texts.
  • I3. Parallel text is used so that content stays fixed and only the factor under study varies, which isolates its effect cleanly.
  • I4. The cost metric is compared against published accuracy (IrokoBench/AfroBench) to test whether costly inputs also tend to be the ones answered badly.
  • I5. A manifest stores pinned versions, a config hash, price snapshots, and an offline reproduce command that needs no keys or downloads.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): the en tokenizer could record in its trace whenever it drops to a fallback splitting path, so tagger and lemma statistics can be compared across runs.
  • I2 (learn): parser and frame evaluations could be repeated on a second held-out text set, treating low cross-set agreement as a warning that a score depends on the content.
  • I3 (back): feeding paraphrase pairs through the text → graph → text compressor should give near-identical meaning graphs, and graph-size ratios between paraphrases could flag wording that bloats or fragments the graph.
  • I4 (answer): logging per-question pipeline cost (tree size, frames, world entities) next to correctness on school tests and fables could reveal whether failures cluster on inputs that are heavy to process.
  • I5 (learn): every silver-data and training run could ship such a manifest plus an offline self-check, so evaluation numbers can be regenerated and audited exactly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Olaoye Anthony Somide. 2026. The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs. arXiv:2606.24460.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2606.24460 · arXiv 2606.24460

@misc{somide2026african,
  title = {{The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs}},
  author = {Olaoye Anthony Somide},
  year = {2026},
  doi = {10.48550/arxiv.2606.24460},
  eprint = {2606.24460},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.24460}
}

References

11 references, 7 of them with a card of ours. ↗ leads to the work.

  1. Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D. R., Smith, N. A., & Tsvetkov, Y. (2023). Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 9904–9923). Association for Computational Linguistics. https://aclanthology.org/2023.emnlp-main.614/ (arXiv: https://arxiv.org/abs/2305.13707) ↗
  2. Adelani, D. I., Ojo, J., Azime, I. A., Zhuang, J. Y., Alabi, J. O., He, X., Ochieng, M., Hooker, S., Bukula, A., Lee, E.-S. A., Chukwuneke, C. I., Buzaaba, H., Sibanda, B. K., Kalipe, G. K., Mukiibi, J., Kabongo Kabenamualu, S., Yuehgoh, F., Setaka, M., Ndolela, L., Odu, N., Mabuya, R., Muhammad, S. H., Osei, S., Samb, S., Guge, T. K., & Stenetorp, P. (2025). IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 2732–2757). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.naacl-long.139 (arXiv: https://arxiv.org/abs/2406.03368) ↗
  3. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for Datasets. Communications of the ACM, 64(12), 86–92. https://doi.org/10.1145/3458723 ↗
  4. NLLB Team. (2024). Scaling neural machine translation to 200 languages. Nature, 630(8018), 841–846. https://doi.org/10.1038/s41586-024-07335-x ↗
  5. Ndomba, G. E., Mswahili, M. E., & Jeong, Y.-S. (2025). Tokenizers for African Languages. IEEE Access, 13, 1046–1054. https://doi.org/10.1109/ACCESS.2024.3522285 ↗
  6. Ojo, J., Ogundepo, O., Oladipo, A., Ogueji, K., Lin, J., Stenetorp, P., & Adelani, D. I. (2023). AfroBench: How Good are Large Language Models on African Languages? arXiv preprint arXiv:2311.07978. https://arxiv.org/abs/2311.07978 ↗
  7. Ovcharov, V. (2026). The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty. arXiv preprint arXiv:2605.24718. https://arxiv.org/abs/2605.24718 ↗
  8. Petrov, A., La Malfa, E., Torr, P., & Bibi, A. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. In Advances in Neural Information Processing Systems (NeurIPS 2023), Vol. 36 (pp. 36963–36990). https://arxiv.org/abs/2305.15425 ↗
  9. Rust, P., Pfeiffer, J., Vulić, I., Ruder, S., & Gurevych, I. (2021). How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), Vol. 1 (Long Papers) (pp. 3118– 3135). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.243 ↗
  10. Lundin, J. M., Zhang, A., Karim, N., Louzan, H., Wei, V., Adelani, D. I., & Carroll, C. (2026). The Token Tax: Systematic Bias in Multilingual Tokenization. In Proceedings of the 7th Workshop on African Natural Language Processing (AfricaNLP 2026) (pp. 103–112). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.africanlp-main.10 (arXiv: https://arxiv.org/abs/2509.05486) Corpora and datasets Adelani, D. I., Alabi, J. O., Fan, A., Kreutzer, J., Shen, X., Reid, M., Ruiter, D., Klakow, D., Nabende, P., Chang, E., Gwadabe, T. G., Sackey, F., Dossou, B. F. P., Emezue, C. C., Leong, C., Beukman, M., Muhammad, S. H., Jarso, G. D., Yousuf, O., Rubungo, A. N. N., Hacheme, G., Wairagala, E. P., Nasir, M. U., Ajibade, B. A., Ajayi, T. O., Gitau, Y. W., Abbott, J., Ahmed, M., Ochieng, M., Aremu, A., Ogayo, P., Mukiibi, J., Kabore, F. O., Kalipe, G. K., Mbaye, D., Tapo, A. A., Koagne, V. M., Munkoh-Buabeng, E., Wagner, V., Abdulmumin, I., Awokoya, A., Buzaaba, H., Sibanda, B. K., Bukula, A., & Manthalu, S. (2022). A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2022) (pp. 3053–3070). Association for Computational Linguistics. https://aclanthology.org/2022.naacl-main.223 ↗
  11. Adelani, D. I., Liu, H., Shen, X., Vassilyev, N., Alabi, J. O., Mao, Y., Gao, H., & Lee, E.-S. A. (2024). SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 226–245). Association for Computational Linguistics. https://aclanthology.org/2024.eacl-long.14 (arXiv: https://arxiv.org/ abs/2309.07445) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/somide-2026-african-language-tax-quantifying-cost/