Elamur Research⌕
☰
home / papers / 2024 / NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning

NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning

Paper · 2024 · Eli Schwartz, Leshem Choshen, Joseph Shtok et al.

Summary

Causal LLMs read numbers left to right, so a digit's place value only becomes clear once the whole number has been seen. The authors argue this contributes to weak arithmetic and propose NumeroLogic: every number gets a digit-count prefix, e.g. "42" becomes "<sn>2<mn>42<en>". For decimals, the prefix gives digit counts for both the integer and fractional parts. The change needs no architectural modification. Regex pre- and post-processing does the conversion, and the prefix is stripped from generated output. When generating, the model must first commit to a digit count, which works like a short chain of thought.

The method was tested in three settings:

  • NanoGPT trained from scratch on five arithmetic tasks
  • Llama2-7B fine-tuned with LoRA on harder arithmetic, both integer and floating-point
  • Continued self-supervised training of Llama2-7B, evaluated on MMLU

The encoding improved accuracy wherever the baseline was not already saturated, and it also raised MMLU scores. This suggests a cheap formatting change can strengthen numerical handling in general language modeling, beyond narrow task-specific fixes.

Key points

  • NanoGPT (character-level, joint multi-task training): NumeroLogic outperformed plain numbers on every task.
    • Addition: 88.37% → 99.96%
    • Subtraction: 73.76% → 97.20%
    • Multiplication: 13.81% → 28.94%
    • Sine and square root: roughly 4-point gains each
  • Llama2-7B (LoRA rank 8, with the expanded embedding and output layers trained in full rank):
    • 5-digit integer addition reached 100%.
    • Floating-point 3-digit multiplication rose from 24.73% to 31.03%.
    • Square root rose from 13.00% to 17.16%.
  • Encoding variants (3-digit integer multiplication):
    • Full NumeroLogic: 35.33%
    • Prefix without "<en>": 34.93%
    • Plain numbers: 34.20%
    • Separate tokens per digit length, such as "<3digitnumber>": 33.56%, which is below plain
  • Is it just the extra tokens?
    • Replacing the added tokens with filler whitespace gave no benefit over plain (24.37% vs 24.73%).
    • Placing the whitespace at random positions, following Shen et al., helped somewhat (27.76%) but stayed below NumeroLogic (31.03%).
    • This indicates the gain comes from the place-value information, not extra compute.
  • Limitations:
    • Numbers take more tokens, and pre- and post-processing add steps.
    • No models larger than 7B were tested, and there was no full pretraining from scratch.
    • Evaluation used English-only benchmarks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States274 works · 82,013 citations🏢IBMArmonk · United States163 works · 73,139 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Eli Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky and Assaf Arbelle. 2024. NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.12 · arXiv 2404.00459

@inproceedings{schwartz2024numerologic,
  title = {{NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning}},
  author = {Eli Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky and Assaf Arbelle},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.12},
  eprint = {2404.00459},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.12}
}

References

19 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.
  2. Philip Gage. 1994. A new algorithm for data compression. The C Users Journal, 12(2):23–38. find
  3. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021a. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR).
  4. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).
  5. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  6. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  7. Andrej Karpathy. 2022. Nanogpt. https://github.com/karpathy/nanoGPT. ↗
  8. Jeonghwan Kim, Giwon Hong, Kyung-min Kim, Junmo Kang, and Sung-Hyon Myaeng. 2021. Have you seen that number? investigating extrapolation in question answering models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7031–7037, Online and Punta Cana, Do… find
  9. Nayoung Lee, Kartik Sreenivasan, Jason D. Lee, Kangwook Lee, and Dimitris Papailiopoulos. 2024. Teaching arithmetic to small transformers. In The Twelfth International Conference on Learning Representations.
  10. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  11. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306… find
  12. James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. 2023. Improving image generation with better captions. Computer Science. https://cdn.openai. com/papers/dall-e-3. pdf, 2(3):8. ↗
  13. Jacob Pfau, William Merrill, and Samuel R Bowman. 2024. Let’s think dot by dot: Hidden computation in transformer language models. arXiv preprint arXiv:2404.15758. find
  14. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
  15. Ruoqi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, and Yi Zhang. 2023. Positional description matters for transformers arithmetic. arXiv preprint arXiv:2311.14737.
  16. Aaditya K Singh and DJ Strouse. 2024. Tokenization counts: the impact of tokenization on arithmetic in frontier llms. arXiv preprint arXiv:2402.14903.
  17. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  18. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mos-quera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, et al. 2023. Findings of the babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyLM Chal… find
  19. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schwartz-2024-numerologic-number-encoding-enhanced-llms/