Elamur Research⌕
☰
home / papers / 2024 / Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models

Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models

Paper · 2024 · Seungduk Kim, Seungtaek Choi and Myeongho Jeong

Summary

English-centric open LLMs handle Korean poorly and wastefully. Their tokenizers split Korean text into many byte-level pieces, which raises cost and latency and shortens the usable context. Adding new tokens is usually thought to require enormous amounts of retraining. The authors propose EEVE, a vocabulary-expansion recipe with two parts:

  • Subword-based initialization. Each new token's input embedding starts as the mean of the embeddings of its constituent subwords. Its output embedding is copied from its first subword.
  • Staged training with parameter freezing. Training starts with only the new input embeddings unfrozen. Later stages progressively open the output embeddings and eventually the full model, with QLoRA used in the full-parameter stage.

They applied this to SOLAR-10.7B and Phi-2 using only about 2 billion tokens of Korean-centric pre-training, producing the EEVE-Korean model family. On KoBEST tasks run through lm-evaluation-harness, the models beat comparable open Korean pre-trained LLMs while keeping their English scores intact. As of January 2024, the 10.8B model was the top pre-trained model on the Open Ko-LLM Leaderboard. The work matters because it shows that cheap, targeted vocabulary expansion can extend strong English models to other languages.

Key points

  • Tokenizer expansion: 8,960 Korean tokens, each seen at least 6,000 times in the corpus, were added on top of the original vocabulary. This gives 40,960 tokens for EEVE-Korean-10.8B-v1.0. In the paper's example, a Korean sentence drops from 26 tokens to 9, roughly a threefold saving.
  • Released models (on Hugging Face under yanolja):
    • EEVE-Korean-10.8B-v1.0, based on SOLAR-10.7B
    • EEVE-Korean-2.8B-v1.0, based on Phi-2
    • Instruction-tuned EEVE-Korean-Instruct variants
  • Benchmarks:
    • Korean: KoBEST (BoolQ, COPA, WiC, HellaSwag, SentiNeg)
    • English: BoolQ, COPA and HellaSwag
  • Comparison with OPEN-SOLAR-KO-10.7B: This model shares the SOLAR base and edges out the EEVE Instruct variant on Korean. However, it loses English accuracy relative to SOLAR-10.7B-v1.0, whereas EEVE does not. A similar pattern holds at small scale against phi-2-ko-v0.1.
  • Effect of preference tuning:
    • English-only tuning can hurt Korean: Korean BoolQ falls from 0.5242 to 0.5157 going from Llama-2-7b to Llama-2-7b-chat.
    • Korean instruction tuning of EEVE did not degrade English and even slightly improved it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Seungduk Kim, Seungtaek Choi and Myeongho Jeong. 2024. Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models. arXiv:2402.14714.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.14714 · arXiv 2402.14714

@misc{kim2024efficient,
  title = {{Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models}},
  author = {Seungduk Kim, Seungtaek Choi and Myeongho Jeong},
  year = {2024},
  doi = {10.48550/arxiv.2402.14714},
  eprint = {2402.14714},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.14714}
}

References

27 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. find
  2. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. The falcon series of open language models. arXiv preprint arXiv:2311.16867.
  3. Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et al. 2023. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312.15166.
  4. Anthropic. 2023. Model card and evaluations for claude models. Anthropic technical Report. find
  5. Ildoo Kim, Gunsoo Han, Jiyeon Ham, and Woonhyuk Baek. 2021. Kogpt: Kakaobrain korean(hangul) generative pretrained transformer. find
  6. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
  7. L. Junbum. 2024. Solar-ko-10.7b. find
  8. Huayang Li, Tian Lan, Zihao Fu, Deng Cai, Lemao Liu, Nigel Collier, Taro Watanabe, and Yixuan Su. 2023a. Repetition in repetition out: Towards understanding neural text degeneration from the data perspective. In Thirty-seventh Conference on Neural Information Processing Systems. find
  9. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023b. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463.
  10. Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, "Teknium", and Nathan Hoos. 2023. Slimorca dedup: A deduplicated subset of slimorca. find
  11. Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  12. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707.
  13. MosaicML NLP Team et al. 2023b. Introducing mpt-30b: Raising the bar for open-source foundation models. find
  14. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  15. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  16. Charles Welch, Rada Mihalcea, and Jonathan K Kummerfeld. 2020. Improving low compute language modeling with in-domain embedding initialisation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8625–8634. find
  17. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  18. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800. find
  19. Chanjun Park, Hwalsuk Lee, Hyunbyung Park, Hyeonwoo Kim, Sanghoon Kim, Seonghwan Cho, Sunghun Kim, and Sukyung Lee. 2023. Open ko-llm leaderboard. find
  20. Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, and Alham Fikri Aji. 2023. Multilingual large language models are not (yet) code-switchers. arXiv preprint arXiv:2305.14235.
  21. Aleksandar Petrov, Emanuele La Malfa, Philip HS Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. arXiv preprint arXiv:2305.15425.
  22. Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llama beyond english: An empirical study on language capability transfer. arXiv preprint arXiv:2401.01055. find
  23. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
  24. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685.
  25. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290.
  26. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. find
  27. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023a. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kim-2024-efficient-effective-vocabulary-expansion-towards/