Elamur Research⌕
☰
home / papers / 2026 / How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?

How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?

Paper · 2026 · Atsuki Yamaguchi, Aline Villavicencio and Nikolaos Aletras

Summary

Tokenizers built mainly for English split text in many other languages into far more pieces. This makes generation slower and more expensive for non-English users. A common fix is to add target-language tokens to the vocabulary, but earlier work used gigabytes of data and randomly initialized the new embeddings. This paper asks whether that approach still works with only about 30K sentences (~0.01GB) of target text, and which recipe suits such scarce data. The authors adapt Llama2 (7B), Llama3 (8B) and Gemma2 (9B) to ten typologically diverse languages. They compare ways of initializing new tokens (Random, FOCUS, Mean, Merge, Align) and several continual pre-training (LAPT) setups. Evaluation covers machine translation, summarization, multiple-choice reading comprehension and GMMLU. The standard high-resource recipe turns out to be suboptimal. Better results come from heuristic initialization based on both tokenizers, tuning only the bottom two and top two layers, a multi-token prediction objective, and short training sequences that allow more updates. The adapted models decode faster and stay close to the source model, especially on generation tasks. This offers a low-cost adaptation route for poorly resourced languages.

Key points

  • Setup: about 30K sentences per language (≈0.01GB, up to roughly 5M tokens). Languages are Arabic, Burmese, German, Greek, Hindi, Japanese, Sinhala, Swahili, Telugu and Thai.
  • On the FLORES-200 dev set, Burmese, Sinhala and Telugu are split most heavily. They need at least 6.8x as many tokens as English with every model tested.
  • Recommended recipe: heuristic initialization (e.g., Mean, Merge), training only the top and bottom two layers (2×2 LS), multi-token prediction (Gloeckle et al. 2024), and short input sequences.
  • Vocabulary replacement performs much worse than expansion in this low-data regime. On Telugu MC it reaches 34 points, against 74 for Source and 60 for expansion with Mean. Its only advantage is a larger speedup in some cases (e.g., 2.95x on Burmese MC).
  • Limitations:
    • Models go up to 9B parameters only.
    • Merge assumes a BPE tokenizer.
    • There is no comparison with MaLA-500 or EMMA-500.
    • The training-free ElChat method only partly recovers classification losses and needs an instruction-tuned source model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SheffieldSheffield · United Kingdom113 works · 21,542 citations🔬The Alan Turing InstituteLondon · United Kingdom29 works · 4,368 citations🎓University of ExeterExeter · United Kingdom12 works · 203 citations🎓Universidade Federal do Rio Grande do NorteNatal · Brazil2 works · 37 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Atsuki Yamaguchi, Aline Villavicencio and Nikolaos Aletras. 2026. How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.a.581

@article{yamaguchi2026how,
  title = {{How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?}},
  author = {Atsuki Yamaguchi, Aline Villavicencio and Nikolaos Aletras},
  journal = {Computational Linguistics},
  year = {2026},
  doi = {10.1162/coli.a.581},
  url = {https://doi.org/10.1162/coli.a.581}
}

References

17 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abbasi, Mohammad Amin, Arash Ghafouri, Mahdi Firouzmandi, Hassan Naderi, and Behrouz Minaei Bidgoli. 2023. PersianLLaMA: Towards building first Persian large language model. arXiv, abs/2312.15713. https://doi.org/10 .21203/rs.3.rs-3789059/v1 Agrawal, Sudhanshu, Wonseok Jeon, and Mingu Lee. 2024. Ada… — [arXiv:2312.15713](https://arxiv.org/abs/2312.15713) find
  2. Ahia, Orevaoghene, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? Tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 990…
  3. Bostrom, Kaj and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617–4624. https://doi.org/10.18653/v1/2020 .findings-emnlp.414 Cahyawijaya, Samuel, Holy Lovenia, Fajri Koto, Rifki P… ↗
  4. Dagan, Gautier, Gabriel Synnaeve, and Baptiste Roziere. 2024. Getting the most out of your tokenizer for pre-training and domain adaptation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 9784–9805.
  5. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, et al. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv, abs/2501.12948. https://doi.org/10 .48550/arXiv.2501.12948 Dobler, Konsta…
  6. Downey, C. M., Terra Blevins, Nora Goldfine, and Shane Steinert-Threlkeld. 2023. Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages. In Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL), pages 268–281. https://doi.org /10.1… ↗
  7. Fujii, Kazuki, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual pre-training for cross-lingual LLM adaptation: Enhancing Japanese language capabilities. In Proceedings of the First Conference on Langu…
  8. Fujii, Takuro, Koki Shibata, Atsuki Yamaguchi, Terufumi Morishita, and Yasuhiro Sogawa. 2023. How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics… find
  9. Groeneveld, Dirk, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. 2024. OLMo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics… find
  10. Hasan, Tahmid, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-Sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, p… find
  11. Hofmann, Valentin, Hinrich Schuetze, and Janet Pierrehumbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 3… find
  12. Huang, Kaixuan, Xudong Guo, and Mengdi Wang. 2025. SpecDec++: Boosting speculative decoding via adaptive candidate lengths. In Proceedings of the Second Conference on Language Modeling. Ji, Shaoxiong, Zihao Li, Jaakko Paavola, Indraneil Paul, Hengyu Luo, and Jörg Tiedemann. 2025. Massively multilin… find
  13. Moi, Anthony and Nicolas Patry. 2023. HuggingFace’s Tokenizers. https://github.com/huggingface/tokenizers Muller, Benjamin, Antonios Anastasopoulos, Benoı̂t Sagot, and Djamé Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models.… ↗
  14. Curran Associates, Inc. Petrov, Aleksandar, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. In Advances in Neural Information Processing Systems, volume 36, pages 36963–36990.
  15. Pipatanakul, Kunat, Phatrasek Jirabovonvisut, Potsawee Manakul, Sittipong Sripaisarnmongkol, Ruangsak Patomwong, Pathomporn Chokchainant, and Kasima Tharnpipitchai. 2023. Typhoon: Thai large language models. arXiv, abs/2312.13951. https://doi.org/10.48550/arXiv.2312.13951 Popović, Maja. 2015. chrF… ↗
  16. Touvron, Hugo, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. LLaMA 2: Open foundation and fine-tuned chat models. arXiv, abs/2307.09288. https://doi.org/10.48550/arXiv.2307.09288 Warstadt… ↗
  17. Yamaguchi, Atsuki, Aline Villavicencio, and Nikolaos Aletras. 2024. An empirical study on cross-lingual vocabulary adaptation for efficient language model inference. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6760–6785. https://doi.org/10.18653/v1/2024 .findings-… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yamaguchi-2026-how-can-we-effectively-expand/