Elamur Research⌕
☰
home / papers / 2025 / Language Models Use Trigonometry to Do Addition

Language Models Use Trigonometry to Do Addition

Paper · 2025 · Subhash Kantamneni and Max Tegmark

Summary

Large language models handle arithmetic reasonably well, but it remains unclear what happens inside them when they add two numbers. Kantamneni and Tegmark reverse engineer integer addition in three mid-sized models: GPT-J, Pythia-6.9B and Llama3.1-8B. They use a dataset of all 10,000 problems a + b with operands from 0 to 99, so every answer fits in a single token. Fourier analysis and PCA of hidden states show that each number is encoded by a linear component plus several periodic (circular) components. The authors call this a generalized helix and fit it to the residual stream with regression. Causal interventions then show that the models apply a variant of the "Clock" algorithm. The helices of both operands are combined into a helix for their sum, and later layers decode that helix into output logits. The study traces this process down to MLPs, attention heads and individual neurons. It is presented as the first explanation of an LLM's math skill that links internal number representations to the computation performed on them.

Key points

  • Accuracy on the 10,000 addition problems: GPT-J 80.5%, Pythia-6.9B 77.2%, Llama3.1-8B 98.0%. The main analysis uses GPT-J because its plain (non-gated) MLP makes individual neurons easier to interpret.
  • The helix fit uses Fourier periods T = [2, 5, 10, 100] plus one linear direction. Neuron preactivations show these same periods.
  • In GPT-J, MLPs 14–18 mostly read the operand helices and build the sum. MLPs 19–27 mostly turn the a + b helix into the answer logits.
  • The helix representation also matters causally for subtraction, multiplication, integer division and modular arithmetic.
  • Limitations:
    • The exact trigonometric step that produces the sum helix could not be isolated.
    • The sum helix plays a weaker causal role in Llama3.1-8B, possibly because of its gated MLPs.
    • Models that tokenize digit by digit, such as Gemma-2-9B, likely need different algorithms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Subhash Kantamneni and Max Tegmark. 2025. Language Models Use Trigonometry to Do Addition. arXiv:2502.00873.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.00873 · arXiv 2502.00873

@misc{kantamneni2025language,
  title = {{Language Models Use Trigonometry to Do Addition}},
  author = {Subhash Kantamneni and Max Tegmark},
  year = {2025},
  doi = {10.48550/arxiv.2502.00873},
  eprint = {2502.00873},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.00873}
}

References

38 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., and Yin, W. Large language models for mathematical reasoning: Progresses and challenges, 2024. URL https://arxiv.org/abs/2402.00157. ↗
  2. Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders, 2024. URL https://arxiv.org/abs/2406.04093. ↗
  3. Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397–2430. PMLR, 20… find
  4. Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., Chen, E., Gunning, A., Olsson, C. F., Denain, J.-S., Ho, A., de Oliveira Santos, E., Järviniemi, O., Barnett, M., Sandler, R., Vrzala, M., Sevilla, J., Ren, Q., Pratt, E., Levine, L., Barkley, G., Stewart, N., Grechuk, B., Grechuk, T., Enugandla,… — FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
  5. Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-Dodds, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T.,… find
  6. Goldowsky-Dill, N., MacLeod, C., Sato, L., and Arora, A. Localizing model behavior with path patching, 2023. URL https://arxiv.org/abs/2304.05969. ↗
  7. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. ↗
  8. Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., a… find
  9. Hanna, M., Liu, O., and Variengien, A. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum? id=p4PckNQR8k. ↗
  10. Heimersheim, S. and Nanda, N. How to use and interpret activation patching, 2024. URL https://arxiv.org/ abs/2404.15255. — [arXiv:2404.15255](https://arxiv.org/abs/2404.15255) ↗
  11. Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., and Olah, C. Toy models of superposition. Transformer Circuits Thread, 2022. URL https://transformer-cir… find
  12. Huben, R., Cunningham, H., Smith, L. R., Ewart, A., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=F76bwRSLeK. ↗
  13. Engels, J., Michaud, E. J., Liao, I., Gurnee, W., and Tegmark, M. Not all language model features are linear, 2024. URL https://arxiv.org/abs/2405. 14860. ↗
  14. Kramár, J., Lieberum, T., Shah, R., and Nanda, N. Atp*: An efficient and scalable method for localizing llm behaviour to components, 2024. URL https://arxiv.org/ abs/2403.00745. ↗
  15. Levy, A. A. and Geva, M. Language models encode numbers using digit representations in base 10, 2024. URL https://arxiv.org/abs/2410.11781. nostalgebraist. interpreting GPT: the logit lens — LessWrong — lesswrong.com. https://www.lesswrong. com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-len… ↗
  16. Liu, H., Dai, Z., So, D., and Le, Q. V. Pay attention to MLPs. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/ forum?id=KBnXrODoBW. ↗
  17. Olah, C. and Jermyn, A. What is a linear representation? what is a multidimensional feature?, 2024. URL https://transformer-circuits. pub/2024/july-update/index.html\# linear-representations. find
  18. Liu, Z., Kitouni, O., Nolte, N., Michaud, E. J., Tegmark, M., and Williams, M. Towards understanding grokking: An effective theory of representation learning. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.… ↗
  19. Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in. — [doi:10.23915/distill.00024.001](https://doi.org/10.23915/distill.00024.001) ↗
  20. Makelov, A., Lange, G., and Nanda, N. Towards principled evaluations of sparse autoencoders for interpretability and control. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024. URL https://openreview.net/forum?id=MHIX9H8aYF. ↗
  21. Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., and Mueller, A. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models, 2024. URL https://arxiv.org/abs/2403. 19647. ↗
  22. Meng, K., Bau, D., Andonian, A. J., and Belinkov, Y. Locating and editing factual associations in GPT. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum? id=-h6WAS6eE4. ↗
  23. Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McC… — In-context Learning and Induction Heads
  24. OpenAI. URL — The Linear Representation Hypothesis and the Geometry of Large Language Models https://openai.com/index/ learning-to-reason-with-llms. Park, K., Choe, Y. J., and Veitch, V. The linear representation hypothesis and the geometry of large language models. In Causal Representation Learning Workshop at NeurIPS 2023, 2023. URL https://openreview. net/forum?id=T0PoOJg8cK. ↗
  25. Rajamanoharan, S., Conmy, A., Smith, L., Lieberum, T., Varma, V., Kramár, J., Shah, R., and Nanda, N. Improving dictionary learning with gated sparse autoencoders, 2024. URL https://arxiv.org/abs/2404.16014. ↗
  26. Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id=9XFSbDPmdW. ↗
  27. Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., et al. Gemma 2: Improving open language models at a practical size, 2024. find
  28. Nanda, N., Rajamanoharan, S., Kramar, J., and Shah, R. Fact finding: Attempting to reverse-engineer factual recall on the neuron level, Dec 2023b. URL https://www.alignmentforum. Satpute, A., Gießing, N., Greiner-Petter, A., Schubotz, M., Teschke, O., Aizawa, A., and Gipp, B. Can org/posts/iGuwZTHWb… — [doi:10.1145/3626772.3657945](https://doi.org/10.1145/3626772.3657945) ↗
  29. Stolfo, A., Belinkov, Y., and Sachan, M. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. pp. 7035–7052, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.435. URL https://aclanthology.org/2023.emnlp-… ↗
  30. Zhu, F., Dai, D., and Sui, Z. Language models encode the value of numbers linearly. In Rambow, O., Wanner, L., Apidianaki, M., Al-Khalifa, H., Eugenio, B. D., and Schockaert, S. (eds.), Proceedings of the 31st International Conference on Computational Linguistics, pp. 693–709, Abu Dhabi, UAE, Januar…
  31. Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, T. Scaling mon… find
  32. Zlokapa, A., Tan, A. K., Martyn, J. M., Fiete, I. R., Tegmark, M., and Chuang, I. L. Fault-tolerant neural networks from biological error correction codes. Phys. Rev. E, 110:054303, Nov 2024. doi: 10.1103/PhysRevE.110. 054303. URL https://link.aps.org/doi/10. 1103/PhysRevE.110.054303. — [doi:10.1103/physreve.110](https://doi.org/10.1103/physreve.110) ↗
  33. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, v…
  34. Wang, B. and Komatsuzaki, A. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax, May 2021. ↗
  35. Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=NpsVSN6o4ul. ↗
  36. Yip, C. H., Agrawal, R., Chan, L., and Gross, J. Modular addition without black-boxes: Compressing explanations of mlps that compute numerical integration, 2024. find
  37. Zhong, Z., Liu, Z., Tegmark, M., and Andreas, J. The clock and the pizza: Two stories in mechanistic explanation of neural networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=S5wmbQc1We. ↗
  38. Zhou, T., Fu, D., Sharan, V., and Jia, R. Pre-trained large language models use fourier features to compute addition. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=i4MutM2TZb. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kantamneni-2025-language-models-use-trigonometry-addition/