Elamur Research⌕
☰
home / papers / 2026 / Language Models Learn Universal Representations of Numbers and Here's Why You Should Care

Language Models Learn Universal Representations of Numbers and Here's Why You Should Care

Paper · 2026 · Michal Štefánik, Timothee Mickus, Marek Kadlčík et al.

Summary

The paper asks how consistently large language models encode numbers. Earlier studies found sine-like number embeddings but did not measure how closely they match across models, and they did not use natural-language contexts. The authors examine eight open checkpoints from the OLMo 2, Llama 3 and Phi 4 families. They compare number embeddings with Representational Similarity Analysis and with overlap of dominant Fourier frequencies after PCA. They also introduce param-sin, a probe that models each latent feature as a single sine wave with learnable frequency, amplitude and phase. They find that number representations are nearly universal: the same periodic structure appears across families, layers and many setups, and numeric values can be decoded accurately throughout the network. Probes trained on real text generalize, while probes trained on templated synthetic data do not. Generic tools such as TunedLens substantially underestimate the information that is present. Steering activations toward cleaner sinusoidal structure reduces some arithmetic errors. This suggests that interpretability tools should be tailored to specific data types.

Key points

  • Cross-model agreement: Number tokens show higher cross-model RSA than a baseline of 1000 random shared word pieces. Among the top k = 63 Fourier frequencies, number embeddings agree perfectly across all model pairs, which does not hold for random tokens.
  • Probe accuracy: Both sinusoidal probes exceed 80% accuracy in all but two layer/model cases and exceed 90% in most of them. A probe trained on one layer also transfers to other layers.
  • Arithmetic steering: Interventions guided by the probes on multiplication and division correct a share of wrong predictions in OLMo 2 (1B–13B), Llama 3 (1B–8B) and Phi-4. This links the precision of sinusoidal structure to output accuracy.
  • Ordinal data: param-sin decodes days of the week (n=7), months (n=12) and the numerals "one"–"ten" under leave-one-out evaluation. Its remaining errors fall only on the first item (e.g., Monday, January).
  • Code and limitations: Code and the probe are released at github.com/prompteus/numllama. The study covers only English-centric models up to 30B parameters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ludwig-Maximilians-Universität MünchenMunich · Germany107 works · 56,478 citations🎓University of HelsinkiHelsinki · Finland50 works · 5,318 citations🎓IT University of CopenhagenCopenhagen · Denmark27 works · 2,406 citations🎓Munich Center for Machine LearningMunich · Germany57 works · 1,094 citations🎓Université de LorraineNancy · France20 works · 945 citations🔬Analyse et Traitement Informatique de la Langue FrançaiseNancy · France6 works · 390 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Michal Štefánik, Timothee Mickus, Marek Kadlčík, Bertram Højer, Michal Spiegel, Raúl Vázquez, Aman Sinha, Josef Kuchař, Philipp Mondorf and Pontus Stenetorp. 2026. Language Models Learn Universal Representations of Numbers and Here's Why You Should Care. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1415

@inproceedings{stefanik2026language,
  title = {{Language Models Learn Universal Representations of Numbers and Here's Why You Should Care}},
  author = {Michal Štefánik, Timothee Mickus, Marek Kadlčík, Bertram Højer, Michal Spiegel, Raúl Vázquez, Aman Sinha, Josef Kuchař, Philipp Mondorf and Pontus Stenetorp},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1415},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1415}
}

References

27 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 oth… — Phi-4 Technical Report
  2. Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. 2025. Eliciting latent predictions from transformers with the tuned lens. Preprint, arXiv:2303.08112.
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  4. Francois Charton. 2022. Linear algebra with transformers. Transactions on Machine Learning Research.
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  6. Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. 2025. Not all language model features are one-dimensionally linear. In The Thirteenth International Conference on Learning Representations. find
  7. Guhao Feng, Kai Yang, Yuntian Gu, Xinyue Ai, Shengjie Luo, Jiacheng Sun, Di He, Zhenguo Li, and Liwei Wang. 2024. How numerical precision affects mathematical reasoning capabilities of llms. Preprint, arXiv:2410.13857. — [arXiv:2410.13857](https://arxiv.org/abs/2410.13857) find
  8. Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. 2023. Towards revealing the mystery behind chain of thought: A theoretical perspective. In Thirty-seventh Conference on Neural Information Processing Systems. find
  9. Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, Bruno Régaldo-Saint Blancard, Tiberiu Tesileanu, Kyunghyun Cho, and Shirley Ho. 2023. xval: A continuous number encoding for large language m… find
  10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  11. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  12. Marek Kadlčík, Michal Štefánik, Timothee Mickus, Josef Kuchař, and Michal Spiegel. 2025. Pre-trained language models learn remarkably accurate representations of numbers. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26693–26702, Suzhou, China. As… find
  13. Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7035–7052, Singapore. Association… find
  14. Subhash Kantamneni and Max Tegmark. 2025. Language models use trigonometry to do addition. Preprint, arXiv:2502.00873. find
  15. YuHong Sun, Zhangyue Yin, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Hui Zhao. 2024. Benchmarking hallucination in large language models based on unanswerable math word problem. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LRE… find
  16. Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. 2008. Representational similarity analysis - connecting the branches of systems neuroscience. Front. Syst. Neurosci., 2:4. find
  17. Amit Arnold Levy and Mor Geva. 2025. Language models encode numbers using digit representations in base 10. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 385–3… find
  18. Yucheng Li, Frank Guerin, and Chenghua Lin. 2024. Latesteval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):18600– 18607. find
  19. Zenan Li, Zhaoyu Li, Wen Tang, Xian Zhang, Yuan Yao, Xujie Si, Fan Yang, Kaiyu Yang, and Xiaoxing Ma. 2025. Proving olympiad inequalities by synergizing llms and symbolic reasoning. Preprint, arXiv:2502.13834. find
  20. Timothee Mickus, Denis Paperno, and Mathieu Constant. 2022. How to dissect a Muppet: The structure of transformer embedding spaces. Transactions of the Association for Computational Linguistics, 10:981–996.
  21. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2023. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations.
  22. Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. 2025. Arithmetic without algorithms: Language models solve math with a bag of heuristics. Preprint, arXiv:2410.21272. find
  23. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, and 21 others. 2025. 2 ol… find
  24. Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations.
  25. Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu ming Cheung, Xinmei Tian, Xu Shen, and Jieping Ye. 2024. Interpreting and improving large language models in arithmetic calculation. Preprint, arXiv:2409.01659. find
  26. Tianyi Zhou, Deqing Fu, Vatsal Sharan, and Robin Jia. 2024. Pre-trained large language models use fourier features to compute addition. In Advances in Neural Information Processing Systems, volume 37, pages 25120–25151. Curran Associates, Inc. find
  27. Fangwei Zhu, Damai Dai, and Zhifang Sui. 2025. Language models encode the value of numbers linearly. In Proceedings of the 31st International Conference on Computational Linguistics, pages 693–709, Abu Dhabi, UAE. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stefanik-2026-language-models-learn-universal-representations/