Elamur Research⌕
☰
home / papers / 2025 / The Token Tax: Systematic Bias in Multilingual Tokenization

The Token Tax: Systematic Bias in Multilingual Tokenization

Paper · 2025 · Jessica M. Lundin, Ada Zhang, Nihal Karim et al.

Summary

Subword tokenizers split text in morphologically rich, low-resource languages into far more pieces than English. This paper measures what that costs in accuracy and money. The authors evaluate 10 LLMs on AfriMMLU, a multiple-choice benchmark of 9,000 items across five subjects in 16 African languages. For each model and language, they compute fertility (average tokens per word) with that model's own tokenizer and record accuracy. They then fit a linear regression of accuracy on fertility for each model–subject pair. A mixed-effects model selected by AIC adds language-specific intercepts and slopes, which suggests the fertility effect differs by language.

Across every model and subject, higher fertility goes with lower accuracy, and fertility explains a substantial share of the variation. Reasoning models such as DeepSeek and o1 narrow the gap to English but do not close it. Because attention cost grows quadratically with sequence length, doubling the token count roughly quadruples training cost and time. The authors treat tokenization as a structural equity problem. They call for morphology-aware tokenizers, fairer pricing and wider multilingual benchmarks.

Key points

  • Regression slopes range from −0.08 to −0.18, so each extra token per word costs about 8–18 accuracy points. Fertility explains 20–50% of accuracy variance. Examples include Llama-3.1-405B on economics (−0.185, p = 0.002; the table labels it "Microeconomics", although the AfriMMLU subject is macroeconomics) and Qwen-2.5-32B on Geography (−0.155, p = 0.006).
  • On average, African languages score about 25 points below English, and French falls in between. DeepSeek and o1 add 8–12 points on African languages. On Global Facts, they shrink the English–African gap from 25 to 12–14 points.
  • Estimated cost of 2× fertility:
    • Training Llama-3.1-405B rises from $105M to $420M.
    • GPT-4o inference per 1M English-equivalent tokens rises from $5–20 to $10–40.
    • Latency goes from about 2 s to 4 s. The conclusion's claim of a 4× latency increase does not match this figure.
  • The mixed-effects analysis leaves out Amharic because it is less similar to the other languages. The accuracy figures count 17 languages, including Amharic.
  • The authors release public data: AfriMMLU results for all models, including the reasoning models, and MMLU token metrics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓McGill UniversityMontreal · Canada87 works · 11,027 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada54 works · 13,565 citations🔬Canadian Institute for Advanced ResearchToronto · Canada28 works · 16,725 citations🤝Gates FoundationSeattle · United States2 works · 85 citations

Also on the byline: University of San Francisco · Institute for Disease Modeling.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksTokenization & segmentation · 119 works

Cite

Cite as Jessica M. Lundin, Ada Zhang, Nihal Karim, Hamza Louzan, Victor Wei, David Adelani and Cody Carroll. 2025. The Token Tax: Systematic Bias in Multilingual Tokenization. arXiv:2509.05486 · Proceedings of the 7th Workshop on African Natural Language Processing (AfricaNLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.africanlp-main.10 · arXiv 2509.05486

@misc{lundin2025token,
  title = {{The Token Tax: Systematic Bias in Multilingual Tokenization}},
  author = {Jessica M. Lundin, Ada Zhang, Nihal Karim, Hamza Louzan, Victor Wei, David Adelani and Cody Carroll},
  year = {2025},
  doi = {10.18653/v1/2026.africanlp-main.10},
  eprint = {2509.05486},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2026.africanlp-main.10}
}

References

7 references, 4 of them with a card of ours. ↗ leads to the work.

  1. David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba Oluwadara Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Ijeoma Chukwuneke, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson Koffi Kalipe, Jonathan Mukiibi, Salomon Kab… — IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models
  2. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Ste… — Tokenizer Choice For LLM Training: Negligible or Crucial?
  3. Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde. On the computational complexity of self-attention, 2022. URL https://arxiv.org/abs/2209.04881. — [arXiv:2209.04881](https://arxiv.org/abs/2209.04881) ↗
  4. Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https://openreview.net/forum?id=78yDLKi95p. ↗
  5. Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages, 2023b. URL https://arxiv.org/abs/2305.15425. ↗
  6. Makesh Narsimhan Sreedhar, Xiangpeng Wan, Yu Cheng, and Junjie Hu. Local byte fusion for neural machine translation, 2023. URL https://arxiv.org/abs/2205.11490. ↗
  7. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/ 1706.03762. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lundin-2025-token-tax-systematic-bias-multilingual/