Elamur Research⌕
☰
home / papers / 2024 / Getting the most out of your tokenizer for pre-training and domain adaptation

Getting the most out of your tokenizer for pre-training and domain adaptation

Summary

LLM developers rarely tune the tokenizer. They typically borrow one and keep it when adapting a base model to a new domain. Code Llama, for example, still uses the original Llama tokenizer. Dagan, Synnaeve and Rozière examine how three tokenizer design choices affect compression, speed, context length, memory and code-generation quality: the tokenizer's training data, its pre-tokenization regular expression and its vocabulary size.

They trained BPE tokenizers with HuggingFace tokenizers on CCnet, Wikipedia and the Stack, measuring compression with Normalized Sequence Length (NSL) relative to Llama. They then trained 1.5B and 7B code models from scratch or fine-tuned them with new tokenizers, and evaluated them on HumanEval and MBPP.

The main findings:

  • Vocabulary size barely affects coding accuracy.
  • Pre-tokenization strongly influences both compression and accuracy.
  • A pre-trained model's tokenizer can be replaced with little loss once fine-tuning exceeds about 50B tokens.

This gives practitioners a practical way to make domain-adapted models faster and to fit more text into their context window.

Key points

  • Code tokenizers can shorten code sequences by over 40% versus Llama. They gain more than 25% with no accuracy loss when using the GPT-4 regex at a 256k vocabulary.
  • The authors recommend the GPT-4 pre-tokenization pattern. It compresses about 5% better than Punct with similar scores. Identity (no pre-tokenization) compresses most but hurts accuracy.
  • Vocabulary size and HumanEval Pass@1 showed only a weak, non-significant correlation (−0.13, p = 0.87).
  • Llama 2 7B was retrained on 500B code tokens with 80k Punct, GPT-4 and Merged tokenizers. These models reached 30.4–31.2% HumanEval Pass@1, versus 32.1% for Code Llama 7B.
  • The final tokenizers were trained on 70% code and 30% English. Evaluation used held-out sets covering 30 programming languages and 28 natural languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Topics: Language models & pretraining · Tokenization & segmentation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.01035 · arXiv 2402.01035

@misc{dagan2024getting,
  title = {{Getting the most out of your tokenizer for pre-training and domain adaptation}},
  author = {Dagan, Gautier and Synnaeve, Gabriel and Roziere, Baptiste},
  year = {2024},
  doi = {10.48550/arxiv.2402.01035},
  eprint = {2402.01035},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.01035}
}

References

46 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. 01.AI. Yi series models: Large language models. find our card
  2. Hugging Face Model Repository, 2023. URL https://huggingface.co/01-ai/Yi-6B. Accessed: 17/11/2023. ↗
  3. Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397–2430. PMLR, 20… find
  4. Ahmad, W., Chakraborty, S., Ray, B., and Chang, K.-W. Unified pre-training for program understanding and generation. In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), Proceedings of the 2021 Conference of… find our card
  5. Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., Pieler, M., Prashanth, U. S., Purohit, S., Reynolds, L., Tow, J., Wang, B., and Weinbach, S. GPT-NeoX-20B: An open-source autoregressive language model. In Fan, A., Ilic, S., Wolf, T… find our card
  6. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce… find our card
  7. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C.… find our card
  8. Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., Umapathi, L. K., Anderson, C. J., Zi, Y., Lamy-Poirier, J., Schoelkopf, H., Troshin, S., Abulkhanov, D., Romero, M., Lappert, M., Toni, F. D., del Rı́o, B. G., Liu, Q., Bose, S., B… find our card
  9. Chirkova, N. and Troshin, S. Codebpe: Investigating subtok-enization options for large language model pretraining on source code. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. find
  10. OpenReview.net, 2023. URL https://openreview.net/pdf?id=htL4UZ344nF. ↗
  11. Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Étienne Goffinet, Hesslow, D., Launay, J., Malartic, Q., Mazzotta, D., Noune, B., Pannier, B., and Penedo, G. The falcon series of open language models, 2023. find our card
  12. Deci. Introducing decicoder: The new gold standard in efficient and accurate code generation, August 2023. URL https://deci.ai/blog/. Accessed: 2024-01-18. ↗
  13. Anthropic. Introducing claude. Anthropic Blog, March 2023. URL https://www.anthropic.com/index/introducing-claude. Accessed: 17/11/2023. ↗
  14. DeepSeek AI. Deepseek coder: A series of code language models. DeepSeek Coder Official Website, 2023. URL https://deepseekcoder.github.io/. Accessed: 17/11/2023. ↗
  15. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T. (eds… find our card
  16. Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., and Nagarajan, V. Think before you speak: Training language models with pause tokens. CoRR, abs/2310.02226, 2023. doi: 10.48550/ARXIV.2310.02226. URL https://doi.org/10.48550/arXiv.2310.02226. guidance-ai. A guidance language for controlling… ↗
  17. Elsen, E., Odena, A., Nye, M., Taşırlar, S., Dao, T., Hawthorne, C., Moparthi, D., and Somani, A. Releasing Persimmon-8B, 2023. URL https://www.adept.ai/blog/persimmon-8b. ↗
  18. Forsythe, A. Tokenmonster: Ungreedy subword tokenizer and vocabulary trainer for python, go and javascript. https://github.com/alasdairforsythe/ tokenmonster, 2023. ↗
  19. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. CoRR, abs/2310.06825, 2023. doi: 10.4855… find
  20. Kocetkov, D., Li, R., Allal, L. B., Li, J., Mou, C., Ferrandis, C. M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., Bahdanau, D., von Werra, L., and de Vries, H. The stack: 3 TB of permissively licensed source code. CoRR, abs/2211.15533, 2022. doi: 10.48550/ARXIV. 2211.15533. URL https://doi.org… ↗
  21. Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, S., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. find
  22. OpenReview.net, 2023. URL https://openreview.net/pdf?id=hQwb-lbM6EL. ↗
  23. Gee, L., Zugarini, A., Rigutini, L., and Torroni, P. Fast vocabulary transfer for language model compression. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 409–416, Abu Dhabi, UAE, December 2022. Association for Computational Linguisti… find
  24. Gee, L., Rigutini, L., Ernandes, M., and Zugarini, A. Multi-word tokenization for sequence compression. In Wang, M. and Zitouni, I. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 612–621, Singapore, December 2023. Association for C… find
  25. Gowda, T. and May, J. Finding the optimal vocabulary size for neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3955–3964, Online, Nov 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.352. URL https://aclant… find our card
  26. Kudo, T. Subword regularization: Improving neural network translation models with multiple subword candidates. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Lo… find our card
  27. Kudo, T. and Richardson, J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Blanco, E. and Lu, W. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Bruss… find our card
  28. Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O., Aiken, A., and Liang, P. Spoc: Search-based pseudocode to code. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference o… — [doi:10.1016/j.artint.2023.103860](https://doi.org/10.1016/j.artint.2023.103860) find
  29. OpenReview.net, 2023. URL https: J., Patel, S. S., Abulkhanov, D., Zocca, M., Dey, M., //openreview.net/pdf?id=iaYcJKpY2B_. Zhang, Z., Moustafa-Fahmy, N., Bhattacharyya, U., Yu, W., Singh, S., Luccioni, S., Villegas, P., Kunakov, M., Nogueira, R. F., Jiang, Z., and Lin, J. Investigating the Zhdanov,… find our card
  30. Replit Blog, October 2023. URL https: languages. In Rogers, A., Boyd-Graber, J., and //blog.replit.com/replit-code-v1_5. Ac-Okazaki, N. (eds.), Findings of the Association for cessed: 17/11/2023. find
  31. Computational Linguistics: ACL 2023, pp. 5661–5681, Rozière, B., Lachaux, M., Chanussot, L., and Lam-Toronto, Canada, July 2023. Association for Computa-ple, G. Unsupervised translation of programming tional Linguistics. doi: 10.18653/v1/2023.findings-acl. languages. In Larochelle, H., Ranzato, M.,… — [doi:10.18653/v1/2023.findings-acl](https://doi.org/10.18653/v1/2023.findings-acl) find
  32. OpenReview.net, 2019. URL https: neurips.cc/paper/2020/hash/ //openreview.net/forum?id=Bkg6RiCqY7. ed23fbf18c2cd35f8c7f8de44f85c08d-Abstract. html. MosaicML. Mpt-7b-instruct: A model for short-form instruction following. MosaicML Blog, May Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I.… find our card
  33. Rust, P., Pfeiffer, J., Vulić, I., Ruder, S., and Gurevych, I. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natura… find our card
  34. Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ili’c, S., Hesslow, D., Castagn’e, R., Luccioni, A. S., Yvon, F., Gallé, M., Tow, J., Rush, A. M., Biderman, S. R., Webson, A., Ammanamanchi, P. S., Wang, T., Sagot, B., Muennighoff, N., del Moral, A. V., Ruwase, O., Bawden, R., Bekman, S., McMillan-Maj… find
  35. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguist… find our card
  36. Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. CCNet: Extracting high quality monolingual datasets from web crawl data. In Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J.,… find our card
  37. Thawani, A., Pujara, J., Ilievski, F., and Szekely, P. Representing numbers in NLP: a survey and a vision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 644–656, Online, June 2021. Association fo… find
  38. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/ARXIV.2302.13971. URL ht… find our card
  39. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Ho… find
  40. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. Huggingface’s transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771, 2019. URL http://arxiv.org/ abs/1910.03771. ↗ our card
  41. Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., and Raffel, C. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306, 2022. doi: 10. 1162/tacl a 00461. URL https://aclantholo… find
  42. Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., Su, T., Yang, Z., and Tang, J. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Singh, A. K., Sun, Y., Akoglu, L., Gunopulos, D., Yan, X., Kumar, R., Ozc… find our card
  43. ACM, 2023. doi: 10.1145/3580305.3599790. URL https://doi.org/10.1145/3580305.3599790. — [doi:10.1145/3580305.3599790](https://doi.org/10.1145/3580305.3599790) ↗
  44. Zipf, G. K. Human Behavior and the Principle of Least Effort. Addison-Wesley, 1949. find
  45. Zouhar, V., Meister, C., Gastaldi, J. L., Du, L., Sachan, M., and Cotterell, R. Tokenization and the noiseless channel. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023… find
  46. Wang, Y., Wang, W., Joty, S., and Hoi, S. C. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696–8708, Online and Punta Cana, Dominican Republic,… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dagan-2024-getting-most-out-tokenizer-pre/