Elamur Research⌕
☰
home / papers / 2026 / The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty

The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty

Paper · 2026 · Volodymyr Ovcharov

Summary

The paper measures the "tokenizer tax," meaning how many more tokens per word a language needs than English. Content is held fixed by using parallel text throughout. Ten foundation models are covered. Seven are called through AWS Bedrock. Three others (GPT-4o, Gemma 2, DeepSeek V3) are counted with their public tokenizers, a shortcut checked against API counts and found to agree within 1.8%. Fertility is computed on concatenated blocks of about 6,000 characters from SIB-200 in 25 European languages. Ukrainian legal, news and Wikipedia text serve to test whether results change by register. Further experiments cover how words are split into subwords, few-shot SIB-200 classification in four Slavic languages, and the ULP grammar benchmark. English costs about 1.23 tokens per word, while Greek and Maltese reach roughly 3.1. Ukrainian, at 2.66, costs 15–18% more than its Slavic relatives, which the author links to scarcer pre-training data. Model rankings stay the same across registers. Whether few-shot prompting helps depends on the model, not the language. The results inform API budgeting, model routing and tokenizer design, and all measurements are released publicly.

Key points

  • Cost by language family: Romance languages sit at 1.5–1.7 tokens/word, Germanic at 1.7–1.9, Slavic at 2.2–2.5 and Uralic/Baltic at 2.7–3.0. Ukrainian requires about 2.2× the English cost per API call. The author ties this to 2–6× less pre-training data than Polish.
  • Domain invariance: The six locally measured models rank identically on news and Wikipedia text (Spearman ρ = 1.0). The spread between cheapest and most expensive model is 1.68× on news, 1.58× on Wikipedia and 1.61× on legal text. Wikipedia is the most expensive register and news the cheapest.
  • Subword splitting: Qwen3 breaks "відповідальність" into 9 pieces, while Gemma 2 needs only 4. DeepSeek V3 has a 128K vocabulary yet still falls among the high-fertility models. The author concludes that the share of Cyrillic-specific merges matters more than vocabulary size.
  • Few-shot and grammar results: Few-shot prompting lowers Llama 4 Maverick by 8.4 pp on average across the four languages and raises Nemotron by 10.5 pp. Fertility correlates only weakly with ULP grammar accuracy (ρ = −0.43, n = 7). Nemotron does well on classification but scores worst on grammar.
  • Release and cost: The data is published as the HuggingFace dataset overthelex/tokenizer-fertility-map, with scripts in the GitHub repo overthelex/SecondLayer. All experiments together cost $4.05.

Interesting ideas

  • I1. Comparing systems on aligned parallel items separates the effect of the language from the effect of the content, which makes any cross-condition comparison cleaner.
  • I2. Before relying on a cheap local measurement, the author checks it against the expensive ground truth on a small overlap set and reports the gap.
  • I3. A rank correlation across registers tests whether the ordering of systems is stable, which shows whether one benchmark is enough.
  • I4. Checking whether subword splits line up with morpheme boundaries gives a mechanistic explanation for an efficiency gap, beyond reporting a single number.
  • I5. The paper treats the effect of few-shot prompting as a property of each model that must be verified per model and task.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation could use paired items, such as the same fable in original and paraphrased wording, so that score differences reflect phrasing rather than story content.
  • I2 (learn): Silver labels from the witness parsers could be validated against an LLM teacher tier on a small shared sample, with the agreement rate reported before the cheaper source is trusted at scale.
  • I3 (tree): Spearman ρ between perceptron parser and gparse variants across different English treebanks would show whether a single dev set can safely guide model choice.
  • I4 (tokens): The lemmatizer and morphology analyzer could be scored on how well their segmentations match derivational boundaries such as -ness or -ation, which would show where lemmas break.
  • I5 (learn): Each LLM teacher tier should get its own few-shot versus zero-shot check per annotation task, instead of one prompt policy applied to all teachers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksTokenization & segmentation · 119 works

Cite

Cite as Volodymyr Ovcharov. 2026. The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty. arXiv:2605.24718.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2605.24718 · arXiv 2605.24718

@misc{ovcharov2026tokenizer,
  title = {{The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty}},
  author = {Volodymyr Ovcharov},
  year = {2026},
  doi = {10.48550/arxiv.2605.24718},
  eprint = {2605.24718},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.24718}
}

References

22 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abadji, J., Ortiz Suarez, P., Romary, L., and Sagot, B. Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. Proceedings of LREC, pages 4344–4355, 2022. arXiv:2201.06642.
  2. Adelani, D. I., et al. SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects. Proceedings of EACL, pages 226–245, 2024. arXiv:2309.07445.
  3. Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D. R., Smith, N. A., and Tsvetkov, Y. Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models. Proceedings of EMNLP, pages 9904–9923, 2023. arXiv:2305.13707.
  4. Brown, T. B., et al. Language Models are Few-Shot Learners. NeurIPS, 2020. arXiv:2005.14165.
  5. Chaplynskyi, D. Ukrainian Brown Corpus and evaluation of multilingual models. Proceedings of the Second Ukrainian NLP Workshop, 2023. find
  6. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. Unsupervised Cross-lingual Representation Learning at Scale. Proceedings of ACL, pages 8440–8451, 2020. arXiv:1911.02116. find
  7. FrancophonIA. EU Acts in Ukrainian: Multilingual Legal Translation Corpus. HuggingFace Datasets, 2022. find
  8. Galeshchuk, S. ULP: Ukrainian Language Proficiency Benchmark for LLMs. HuggingFace Datasets, 2024. find
  9. Lai, V. D., Ngo, N. T., Veyseh, A. P. B., Man, H., Dernoncourt, F., Bui, T., and Nguyen, T. H. ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning. Findings of EMNLP, 2023. arXiv:2304.05613.
  10. Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What Makes Good In-Context Examples for GPT-3? Proceedings of DeeLIO Workshop at ACL, 2022. arXiv:2101.06804.
  11. Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? Proceedings of EMNLP, 2022. arXiv:2202.12837.
  12. Nguyen, T., Nguyen, H., Lai, V. D., Man, H., Ngo, N. T., Dernoncourt, F., Rossi, R. A., and Nguyen, T. H. CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages. Proceedings of ACL: System Demonstrations, pages 228–237, 2024. arXiv:2309.09400.
  13. Niklaus, J., Chalkidis, I., and Stürmer, M. LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain. Findings of EACL, pages 1721–1734, 2024. arXiv:2301.13126.
  14. Ovcharov, V. Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study. arXiv:2605.14890, 2026. find
  15. Petrov, A., La Malfa, E., Torr, P., and Bibi, A. Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS, 2023. arXiv:2305.15425.
  16. Rust, P., Pfeiffer, J., Vulić, I., Ruder, S., and Gurevych, I. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. Proceedings of ACL, pages 3118–3135, 2021. arXiv:2012.15613.
  17. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS, 2022. arXiv:2201.11903.
  18. Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate Before Use: Improving Few-Shot Performance of Language Models. Proceedings of ICML, 2021. arXiv:2102.09690.
  19. Clark, J. H., Garrette, D., Turc, I., and Wieting, J. CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation. Transactions of the Association for Computational Linguistics, 10:73–91, 2022. arXiv:2103.06874.
  20. Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, N. A. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. Proceedings of ACL, pages 8342–8360, 2020. arXiv:2004.10964. find
  21. Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., and Raffel, C. ByT5: Towards a Token-Free Future with Pre-Trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics, 10:291–306, 2022. arXiv:2105.13626. find
  22. Zheng, B., Dong, L., Huang, S., Singhal, S., Che, W., Liu, T., Song, X., and Wei, F. Allocating Large Vocabulary Capacity for Cross-Lingual Language Model Pre-Training. Proceedings of EMNLP, pages 4684–4694, 2021. arXiv:2109.07306. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ovcharov-2026-tokenizer-tax-across-25-european/