Elamur Research⌕
☰
home / papers / 2023 / xVal: A Continuous Numerical Tokenization for Scientific Language Models

xVal: A Continuous Numerical Tokenization for Scientific Language Models

Paper · 2023 · Siavash Golkar, Mariel Pettee, Michael Eickenberg et al.

Summary

Standard LLMs split numbers into discrete text tokens, so they do not capture magnitude or continuity well. This makes them poorly suited to number-heavy scientific data. The Polymathic AI Collaboration proposes xVal, which represents every number with one shared [NUM] token. That token's learnable embedding is scaled by the actual numeric value. On the output side, a separate scalar number head trained with MSE predicts the value whenever the token head emits [NUM]. Together, these make the transformer an end-to-end continuous function of the numbers in its input. The authors trained GPT-2-style models from scratch on scientific data rendered as text: a global temperature dataset and simulated planetary orbits, plus arithmetic in an appendix. They compared xVal against several text-based encodings, such as P10 and P1000. xVal gave the best overall balance of in-distribution accuracy and out-of-distribution interpolation, and it was the cheapest to compute. It struggled where the target is highly uncertain. The work matters because it points toward text-based foundation models that can handle mixed numeric and textual scientific data.

Key points

  • Encoding: Numbers are parsed out and replaced by [NUM]. Each [NUM] embedding is multiplied by its value, so every number costs exactly one token and the vocabulary grows by only one entry.
  • Wider value ranges: An optional extension adds several [NUM]_i embeddings weighted by tanh at different orders of magnitude. With this extension, the number head is trained with NMSE instead of MSE.
  • Experiments: Models are GPT-2-based and trained from scratch. Tasks are temperature forecasting and planetary-orbit prediction (future timesteps, step size, mass), with arithmetic covered in Appendix A.
  • Results: xVal performed best on next-step temperature prediction, on forecasting planets' final 10 timesteps, and on out-of-distribution interpolation. Text encodings such as P1000 interpolated poorly, and P10 was too costly. All methods failed on planet mass, where xVal did unusually badly; the authors suggest a Gaussian-mixture number head as a fix.
  • Code: Released at github.com/PolymathicAI/xVal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom287 works · 125,174 citations🎓University of Colorado BoulderBoulder · United States76 works · 51,396 citations🔬Centre National de la Recherche ScientifiqueParis · France173 works · 62,617 citations🎓Université Paris-SaclayGif-sur-Yvette · France49 works · 6,355 citations🔬Lawrence Berkeley National LaboratoryBerkeley · United States5 works · 41,967 citations🎓Université Paris CitéParis · France22 works · 1,476 citations🏛️Commissariat à l'Énergie Atomique et aux Énergies AlternativesParis · France12 works · 435 citations🔬Flatiron InstituteNew York · United States2 works · 235 citations

Also on the byline: Astrophysique, Instrumentation et Modélisation.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksTokenization & segmentation · 125 works

Cite

Cite as Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, Bruno Régaldo-Saint Blancard, Tiberiu Tesileanu, Kyunghyun Cho and Shirley Ho. 2023. xVal: A Continuous Numerical Tokenization for Scientific Language Models. arXiv:2310.02989.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.02989 · arXiv 2310.02989

@misc{golkar2023xval,
  title = {{xVal: A Continuous Numerical Tokenization for Scientific Language Models}},
  author = {Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, Bruno Régaldo-Saint Blancard, Tiberiu Tesileanu, Kyunghyun Cho and Shirley Ho},
  year = {2023},
  doi = {10.48550/arxiv.2310.02989},
  eprint = {2310.02989},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.02989}
}

References

37 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring Length Generalization in Large Language Models, 2022.
  2. Ali Borji. A Categorical Archive of ChatGPT Failures, 2023.
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  4. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. ↗
  5. François Charton. Linear Algebra with Transformers, 2022.
  6. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative Pretraining From Pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.… find
  7. Charles Q. Choi. 7 Revealing Ways AIs Fail: Neural Networks can be Disastrously Brittle, Forgetful, and Surprisingly Bad at Math. IEEE Spectrum, 58(10):42–47, 2021. doi: 10.1109/MSPEC.2021.9563958. — [doi:10.1109/mspec.2021.9563958](https://doi.org/10.1109/mspec.2021.9563958) find
  8. Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results, 2014. find
  9. Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and Controllable Music Generation, 2023. find
  10. Stéphane d’Ascoli, Pierre-Alexandre Kamienny, Guillaume Lample, and François Charton. Deep Symbolic Regression for Recurrent Sequences, 2022.
  11. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
  12. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality,…
  13. Simon Frieder, Luca Pinchetti, Alexis Chevalier, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, and Julius Berner. Mathematical capabilities of chatgpt, 2023.
  14. Philip Gage. A New Algorithm for Data Compression. C Users J., 12(2):23–38, feb 1994. ISSN 0898-9788. find
  15. Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? A Case Study of Simple Function Classes. Advances in Neural Information Processing Systems, 2022.
  16. Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying Large Language Model Generalizatio… — Studying Large Language Model Generalization with Influence Functions
  17. Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters, 2023.
  18. Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, Adrian Simmons, Cornel Soci, Saleh Abdalla, Xavier Abellan, Gianpaolo Balsamo, Peter Bechtold, Gionata Biavati, Jean Bidlot, Massimo Bonavita… — [doi:10.1002/qj.3803](https://doi.org/10.1002/qj.3803) find
  19. Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models, 2023.
  20. Chengyue Jiang, Zhonglin Nian, Kaihao Guo, Shanbo Chu, Yinggong Zhao, Libin Shen, and Kewei Tu. Learning Numeral Embeddings, 2020. find
  21. Mingyu Jin, Hua Tang, Chong Zhang, Qinkai Yu, Chengzhi Liu, Suiyuan Zhu, Yongfeng Zhang, and Mengnan Du. Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilities, 2024. find
  22. Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. arXiv preprint arXiv:2210.10749, 2022. find
  23. Tiedong Liu and Bryan Kian Hsiang Low. Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks, 2023.
  24. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show Your Work: Scratchpads for Intermediate Computation with Language Models, 2021.
  25. OpenAI. Gpt-4 technical report, 2023.
  26. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets, 2022.
  27. Guanghui Qin, Yukun Feng, and Benjamin Van Durme. The NLP Task Effectiveness of Long-Range Transformers, 2023. find
  28. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. find
  29. Hanno Rein and David S. Spiegel. IAS15: a fast, adaptive, high-order integrator for gravitational dynamics, accurate to machine precision over a billion orbits. Monthly Notices of the Royal Astronomical Society, 446(2):1424–1437, January 2015. doi: 10.1093/mnras/stu2164. — [doi:10.1093/mnras/stu2164](https://doi.org/10.1093/mnras/stu2164) find
  30. Rein, H. and Liu, S.-F. REBOUND: an open-source multi-purpose N-body code for collisional dynamics. A&A, 537:A128, 2012. doi: 10.1051/0004-6361/201118085. URL https://doi.org/10.1051/0004-6361/201118085. — [doi:10.1051/0004-6361/201118085](https://doi.org/10.1051/0004-6361/201118085) ↗
  31. Dhanasekar Sundararaman, Shijing Si, Vivek Subramanian, Guoyin Wang, Devamanyu Hazarika, and Lawrence Carin. Methods for Numeracy-Preserving Word Embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4742–4753, Online, November 2020. Assoc… find
  32. Alberto Testolin. Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models, 2023. find
  33. Avijit Thawani, Jay Pujara, Pedro A. Szekely, and Filip Ilievski. Representing Numbers in NLP: a Survey and a Vision, 2021.
  34. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  35. Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models. CoRR, abs/2007.06778, 2020. URL https://arxiv.org/abs/2007.06778. ↗
  36. Larry Wasserman. All of Nonparametric Statistics. Springer Science & Business Media, 2006. find
  37. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/golkar-2023-xval-continuous-numerical-tokenization-scientific/