Elamur Research⌕
☰
home / papers / 2024 / Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs

Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs

Summary

The paper asks whether the way numbers are split into tokens affects how well frontier LLMs do arithmetic. GPT-3.5 and GPT-4 use the cl100k_base tokenizer, which has a token for every 1-, 2- and 3-digit string and chunks long numbers into groups of three starting from the left (L2R). By contrast, LLaMa, PaLM and Mistral use single-digit tokens, and older GPT-3 relied on irregular pure BPE splits. The authors ran few-shot addition problems through OpenAI chat models with greedy decoding. They compared the default L2R chunking with right-to-left (R2L) chunking, which they forced at inference time by inserting thousands-separator commas. R2L inputs gave much higher accuracy. The authors trace the L2R failures to misalignment between how the addends and the answer are tokenized, and they show that these errors follow systematic patterns rather than random noise. Asking the model to first rewrite numbers in R2L form recovers most of the lost accuracy. The work shows that tokenization is a real inductive bias that model builders should test deliberately.

Key points

  • Accuracy on the headline addition task (March 2023 checkpoints), L2R vs. comma-enforced R2L:
    • GPT-3.5: 75.6% vs. 97.8%
    • GPT-4: 84.4% vs. 98.9%
    • Overall, R2L gains reach about 20 points.
  • Models tested: gpt-3.5-turbo-0301 (main), gpt-3.5-turbo-0613, gpt-4-0314 and gpt-4-0613. Tokenizers compared: p50k_base (GPT-3) and cl100k_base (GPT-3.5/4).
  • Under L2R, GPT-3.5 shows a strikingly regular error pattern, the "digit 4" error, which suggests flawed but systematic internal computation.
  • A chain-of-thought-style mitigation, where the model repeats the input in R2L format before answering, closes most of the accuracy gap.
  • The L2R/R2L gap narrows in larger models, so smaller or distilled models are more exposed to this bias. Code and results are available at github.com/aadityasingh/TokenizationCounts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Topics Language models & pretraining Tokenization & segmentation

Cite

Cite as Aaditya K. Singh and DJ Strouse. 2024. Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs. arXiv:2402.14903.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.14903 · arXiv 2402.14903

@misc{singh2024tokenization,
  title = {{Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs}},
  author = {Aaditya K. Singh and DJ Strouse},
  year = {2024},
  doi = {10.48550/arxiv.2402.14903},
  eprint = {2402.14903},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.14903}
}

References

41 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anil, C., Wu, Y., Andreassen, A. J., Lewkowycz, A., Misra, V., Ramasesh, V. V., Slone, A., Gur-Ari, G., Dyer, E., and Neyshabur, B. Exploring length generalization in large language models. Neural Information Processing Systems (NeurIPS), 2022. URL https://openreview.net/ forum?id=zSkYVeX7bC4. ↗ our card
  2. URL Deng, Y., Zhang, W., Chen, Z., and Gu, Q. Rephrase and respond: Let large language models ask better questions for themselves. arXiv:2311.04205, 2023. URL https://arxiv.org/abs/2311.04205. ↗
  3. Gage, P. A new algorithm for data compression. C Users Journal, 1994. find
  4. Bostrom, K. and Durrett, G. Byte pair encoding is suboptimal for language model pretraining. Empirical Methods in Natural Language Processing (EMNLP), 2020. URL https://aclanthology.org/ 2020.findings-emnlp.414. ↗ our card
  5. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The Pile: An 800gb dataset of diverse text for language modeling. arXiv:2101.00027, 2020. URL https://arxiv.org/abs/2101.00027. ↗
  6. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… our card
  7. Gemini Team, Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., Silver, D., Petrov, S., Johnson, M., Antonoglou, I., Schrittwieser, J., Glaese, A., Chen, J., Pitler, E., Lillicrap, T., Lazaridou, A., Firat, O., Molloy, J., Isard,… our card
  8. Golkar, S., Pettee, M., Eickenberg, M., Bietti, A., Cranmer, M., Krawezik, G., Lanusse, F., McCabe, M., Ohana, R., Parker, L., Blancard, B. R.-S., Tesileanu, T., Cho, K., and Ho, S. xval: A continuous number encoding for large language models. Neural Information Processing Systems (NeurIPS) AI for S… find
  9. Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., and Nagarajan, V. Think before you speak: Training language models with pause tokens. International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=ph04CRkPdC. ↗
  10. Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K. R., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N., Naik, A., Nam,… find
  11. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021. URL https://openreview.net/forum?id=7Bywt2mQsCe. ↗ our card
  12. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. arXiv:2310.06825, 2023. URL https://a… find
  13. Muffo, M., Cocco, A., and Bertino, E. Evaluating transformer language models on arithmetic operations using number decomposition. Language Resources and Evaluation Conference (LREC), 2022. URL https://aclanthology.org/2022.lrec-1.30. ↗
  14. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Neural Information Processing Systems (NeurIPS), 2022. URL https://openreview.net/ forum?id=e2TBb5y0yFf. ↗ our card
  15. Nogueira, R., Jiang, Z., and Lin, J. Investigating the limitations of transformers with simple arithmetic tasks. arXiv:2102.13019, 2021. URL https://arxiv.org/abs/2102.13019. ↗
  16. Kudo, T. Subword regularization: Improving neural network translation models with multiple subword candidates. Association for Computational Linguistics (ACL), 2018. URL https://aclanthology.org/P18-1007. ↗ our card
  17. Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. Show your work: Scratchpads for intermediate computation with language models. arXiv:2112.00114, 2021. URL https://arxiv.org/abs/2112.00114. ↗ our card
  18. Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., et al. Measuring faithfulness in chain-of-thought reasoning. arXiv:2307.13702, 2023. URL https://arxiv.org/abs/2307.13702. ↗ our card
  19. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bog… find
  20. Lewkowycz, A., Andreassen, A. J., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V. V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., and Misra, V. Solving quantitative reasoning problems with language models. Neural Information Processing Systems (NeurIPS),… our card
  21. Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. Textbooks Are All You Need II: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023. our card
  22. Liu, R., Garrette, D., Saharia, C., Chan, W., Roberts, A., Narang, S., Blok, I., Mical, R., Norouzi, M., and Constant, N. Character-aware models improve visual text rendering. Association for Computational Linguistics (ACL), 2023. URL https://aclanthology.org/ 2023.acl-long.900. ↗
  23. Lundberg, S. The art of prompt design: Prompt boundaries and token healing, 2023. URL https://towardsdatascience.com/the-art-of-prompt-design-prompt-boundaries-and-token-healing-3b2448b0be38. ↗
  24. McCoy, R. T., Yao, S., Friedman, D., Hardy, M., and Griffiths, T. L. Embers of autoregression: Understanding large language models through the problem they Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O’Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pa… our card
  25. Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. Empirical Methods in Natural Language Processing (EMNLP), 2020. URL https://aclanthology.org/2020.emnlp-main.346. ↗ our card
  26. Stolfo, A., Belinkov, Y., and Sachan, M. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. Empirical Methods in Natural Language Processing (EMNLP), 2023. URL https://openreview.net/ forum?id=aB3Hwh4UzP. ↗
  27. Sun, K., Qi, P., Zhang, Y., Liu, L., Wang, W., and Huang, Z. Tokenization consistency matters for generative models on extractive NLP tasks. Empirical Methods in Natural Language Processing (EMNLP), 2023. URL https://aclanthology.org/ 2023.findings-emnlp.887. ↗
  28. Teknium. How did the gpt tokenizer get created?, 2023. URL https://twitter.com/Teknium1/ status/1634667026739527680?s=20. ↗
  29. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv:2302.13971, 2023a. URL https://arxiv.org/abs/2302.13971. ↗ our card
  30. Paster, K. Testing language models on a held-out high school national finals exam. https://huggingface.co/datasets/keirp/ hungarian national hs finals exam, 2023. ↗
  31. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossei… find
  32. Razeghi, Y., Logan IV, R. L., Gardner, M., and Singh, S. Impact of pretraining term frequencies on few-shot numerical reasoning. Empirical Methods in Natural Language Processing (EMNLP), 2022. URL https://aclanthology.org/ 2022.findings-emnlp.59. ↗ our card
  33. Rumbelow, J. and mwatkins. Solidgoldmagikarp (plus, prompt generation), 2023. URL https://www.lesswrong.com/posts/ aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation. ↗
  34. Saxton, D., Grefenstette, E., Hill, F., and Kohli, P. Analysing mathematical reasoning abilities of neural models. International Conference on Learning Representations (ICLR), 2019. URL https://openreview.net/ forum?id=H1gR5iR5FX. ↗ our card
  35. Wang, B. and Komatsuzaki, A. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, 2021. URL https://huggingface.co/EleutherAI/ gpt-j-6b. ↗
  36. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. Association for Computational Linguistics (ACL), 2016. URL https://aclanthology.org/P16-1162. ↗ our card
  37. Wei, J. Sorting a list of words by the second letter, 2023. URL https://x.com/ jasonwei/ status/1661781746759909376?s=20. ↗
  38. Wei, J., Wang, X., Schuurmans, D., Bosma, M., brian ichter, Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. Neural Information Processing Systems (NeurIPS), 2022. URL https://openreview.net/ forum?id= VjQlMeSB J. ↗ our card
  39. Yu, L., Simig, D., Flaherty, C., Aghajanyan, A., Zettlemoyer, L., and Lewis, M. MEGABYTE: Predicting million-byte sequences with multiscale transformers. Neural Information Processing Systems (NeurIPS), 2023. URL https://openreview.net/forum?id=JTmO2V9Xpz. ↗
  40. Zhou, H., Nova, A., Larochelle, H., Courville, A., Neyshabur, B., and Sedghi, H. Teaching algorithmic reasoning via in-context learning. arXiv:2211.09066, 2022. URL https://arxiv.org/abs/2211.09066. ↗
  41. Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., and Nakkiran, P. What algorithms can transformers learn? a study in length generalization. arXiv:2310.16028, 2023. URL https://arxiv.org/abs/2310.16028. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2024-tokenization-counts-impact-tokenization-arithmetic/