Elamur Research⌕
☰
home / papers / 2023 / SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training

SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training

Paper · 2023 · Kazem Meidani, Parshin Shojaee, Chandan K. Reddy et al.

Summary

Deep learning models for mathematics usually handle either symbolic expressions or numeric data, and they are typically trained with supervision for one specific task. SNIP (Symbolic-Numeric Integrated Pre-training) instead learns a shared representation of both modalities. It uses two Transformer encoders, one for equations and one for sampled (x, y) observations. A CLIP-style contrastive objective pulls matching pairs together in embedding space. Pre-training uses about 60 million synthetic equation–data pairs.

The pre-trained embeddings were evaluated in two ways:

  • Symbolic-to-numeric: predicting mathematical properties of functions, where SNIP beats fully supervised baselines, especially with little labeled data.
  • Numeric-to-symbolic (symbolic regression): a decoder is added on top of the numeric encoder, and gradient-free search is run in SNIP's continuous latent space to find accurate yet compact equations.

On SRBench, SNIP is competitive with state-of-the-art methods and produces simpler formulas. The work matters because it shows that cross-modal pre-training improves mathematical understanding and makes equation discovery more interpretable.

Key points

  • Numeric tokenization: values are rounded to four significant digits and split into three tokens (sign, mantissa, exponent), e.g. 5.432 becomes [+, 5432, E-3].
  • Black-box regression (SRBench, 57 datasets): SNIP ranks fourth by median R² and sits on the first Pareto front for accuracy versus complexity. Its mean complexity of 47.52 is lower than Operon (64.95), FEAT (74.18), EPLEX (55.82) and E2E (82.78).
  • Closest competitor, Operon: it fits the data better (R² 0.933 vs SNIP's 0.872) but returns more complex expressions.
  • Latent space optimization ablation (400 synthetic functions): the share of solutions with R² > 0.99 rises from 0.683 without optimization to 0.820 with Grey Wolf Optimizer. Nevergrad's NGOpt and TwoPointsDE optimizers both reach 0.805.
  • Availability: code and model are released in the GitHub repository deep-symbolic-mathematics/Multimodal-Math-Pretraining.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksMathematics · 233 worksComputational Theory and Mathematics · 496 worksPhysics and Astronomy · 70 worksStatistical and Nonlinear Physics · 60 worksModeling and Simulation · 12 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Kazem Meidani, Parshin Shojaee, Chandan K. Reddy and Amir Barati Farimani. 2023. SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training. arXiv:2310.02227.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.02227 · arXiv 2310.02227

@misc{meidani2023snip,
  title = {{SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training}},
  author = {Kazem Meidani, Parshin Shojaee, Chandan K. Reddy and Amir Barati Farimani},
  year = {2023},
  doi = {10.48550/arxiv.2310.02227},
  eprint = {2310.02227},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.02227}
}

References

57 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. A cookbook of self-supervised learning. arXiv preprint arXiv:2304.12210, 2023. — [arXiv:2304.12210](https://arxiv.org/abs/2304.12210) find
  2. Sören Becker, Michal Klein, Alexander Neitz, Giambattista Parascandolo, and Niki Kilbertus. Predicting ordinary differential equations with transformers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th Interna… find
  3. Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascandolo. Neural symbolic regression that scales. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Researc… find
  4. Bogdan Burlacu, Gabriel Kronberger, and Michael Kommenda. Operon c++: An efficient genetic programming framework for symbolic regression. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, GECCO ’20, pp. 1562–1570, New York, NY, USA, 2020. Association for Computing… — [doi:10.1145/3377929.3398099](https://doi.org/10.1145/3377929.3398099) find
  5. Zhonglin Cao, Rishikesh Magar, Yuyang Wang, and Amir Barati Farimani. Moformer: Self-supervised transformer model for metal–organic framework property prediction. Journal of the American Chemical Society, 145(5):2958–2967, 2023. doi: 10.1021/jacs.2c11420. URL https://doi.org/10.1021/jacs.2c11420. PM… — [doi:10.1021/jacs.2c11420](https://doi.org/10.1021/jacs.2c11420) ↗
  6. Francois Charton. Linear algebra with transformers. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=Hp4g7FAXXG. ↗
  7. Miles Cranmer. Interpretable machine learning for science with pysr and symbolicregression. jl. arXiv preprint arXiv:2305.01582, 2023.
  8. Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA…
  9. Stéphane D’Ascoli, Pierre-Alexandre Kamienny, Guillaume Lample, and Francois Charton. Deep symbolic regression for recurrence prediction. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machin… find
  10. Xiao Dong, Xunlin Zhan, Yangxin Wu, Yunchao Wei, Michael C Kampffmeyer, Xiaoyong Wei, Minlong Lu, Yaowei Wang, and Xiaodan Liang. M5product: Self-harmonized contrastive learning for e-commercial multi-modal pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni… find
  11. Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889–898, Melbourne, Australia, jul 2018. Association for Computational Linguistics. doi: 10.18653/v1…
  12. Roger Fletcher. Practical Methods of Optimization. John Wiley & Sons, New York, NY, USA, second edition, 1987. find
  13. Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 946–958, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.acl-mai…
  14. Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, et al. xval: A continuous number encoding for large language models. arXiv preprint arXiv:2310.02989, 2023.
  15. Paul Hager, Martin J Menten, and Daniel Rueckert. Best of both worlds: Multimodal contrastive learning with tabular and imaging data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23924–23935, 2023. find
  16. Samuel Holt, Zhaozhi Qian, and Mihaela van der Schaar. Deep generative symbolic regression. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=o7koEEMA1bR. ↗
  17. Samy Jelassi, Stéphane d’Ascoli, Carles Domingo-Enrich, Yuhuai Wu, Yuanzhi Li, and François Charton. Length generalization in arithmetic transformers. arXiv preprint arXiv:2306.15400, 2023.
  18. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference…
  19. Pierre-Alexandre Kamienny, Stéphane d’Ascoli, Guillaume Lample, and Francois Charton. End-to-end symbolic regression with transformers. In Advances in Neural Information Processing Systems, 2022.
  20. William La Cava, Kourosh Danai, and Lee Spector. Inference of compact nonlinear dynamic models by epigenetic local search. Engineering Applications of Artificial Intelligence, 55:292–306, 2016. ISSN 0952-1976. doi: https://doi.org/10.1016/j.engappai.2016.07.004. URL https://www. sciencedirect.com/sc… — [doi:10.1016/j.engappai.2016.07.004](https://doi.org/10.1016/j.engappai.2016.07.004) ↗
  21. William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabricio de Franca, Marco Virgolin, Ying Jin, Michael Kommenda, and Jason Moore. Contemporary symbolic regression methods and their relative performance. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing System…
  22. Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum? id=S1eZYeHFDS. ↗
  23. Guillaume Lample, Timothee Lacroix, Marie-Anne Lachaux, Aurelien Rodriguez, Amaury Hayat, Thibaut Lavril, Gabriel Ebner, and Xavier Martinet. Hypertree proof search for neural theorem proving. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information…
  24. Mikel Landajuela, Chak Lee, Jiachen Yang, Ruben Glatt, Claudio P. Santiago, Ignacio Aravena, Terrell N. Mundhenk, Garrett Mulcahy, and Brenden K. Petersen. A unified framework for deep symbolic regression. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural… find
  25. Zhenwen Liang, Jipeng Zhang, Lei Wang, Wei Qin, Yunshi Lan, Jie Shao, and Xiangliang Zhang. Mwp-bert: Numeracy-augmented pre-training for math word problem solving. In Findings of NAACL 2022, pp. 997–1009, 2022. find
  26. Jing Liu, Xinxin Zhu, Fei Liu, Longteng Guo, Zijia Zhao, Mingzhen Sun, Weining Wang, Hanqing Lu, Shiyu Zhou, Jiajun Zhang, et al. Opt: Omni-perception pre-trainer for cross-modal understanding and generation. arXiv preprint arXiv:2107.00249, 2021. — [arXiv:2107.00249](https://arxiv.org/abs/2107.00249) find
  27. Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. A survey of deep learning for mathematical reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14605–14631, Toronto, Canada, July 2023. Association for Comput… find
  28. Kazem Meidani and Amir Barati Farimani. Identification of parametric dynamical systems using integer programming. Expert Systems with Applications, 219:119622, 2023. ISSN 0957-4174. doi: https://doi.org/10.1016/j.eswa.2023.119622. — [doi:10.1016/j.eswa.2023.119622](https://doi.org/10.1016/j.eswa.2023.119622) ↗
  29. Sebastian Mežnar, Sašo Džeroski, and Ljupčo Todorovski. Efficient generator of mathematical expressions for symbolic regression. Machine Learning, Sep 2023. ISSN 1573-0565. doi: 10.1007/ s10994-023-06400-2. URL https://doi.org/10.1007/s10994-023-06400-2. — [doi:10.1007/s10994-023-06400-2](https://doi.org/10.1007/s10994-023-06400-2) ↗
  30. Seyedali Mirjalili, Seyed Mohammad Mirjalili, and Andrew Lewis. Grey wolf optimizer. Advances in Engineering Software, 69:46–61, 2014. ISSN 0965-9978. doi: https://doi.org/10. 1016/j.advengsoft.2013.12.007. URL https://www.sciencedirect.com/science/ article/pii/S0965997813001853. find
  31. Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734, 2021. — [arXiv:2111.09734](https://arxiv.org/abs/2111.09734) find
  32. Terrell N. Mundhenk, Mikel Landajuela, Ruben Glatt, Claudio P. Santiago, Daniel faissol, and Brenden K. Petersen. Symbolic regression via deep reinforcement learning enhanced genetic programming seeding. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Infor… find
  33. Randal S. Olson, William La Cava, Patryk Orzechowski, Ryan J. Urbanowicz, and Jason H. Moore. Pmlb: a large benchmark suite for machine learning evaluation and comparison. BioData Mining, 10(1):36, Dec 2017. ISSN 1756-0381. doi: 10.1186/s13040-017-0154-4. URL https://doi. org/10.1186/s13040-017-0154… — [doi:10.1186/s13040-017-0154-4](https://doi.org/10.1186/s13040-017-0154-4) find
  34. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  35. Shuai Peng, Ke Yuan, Liangcai Gao, and Zhi Tang. Mathbert: A pre-trained model for mathematical formula understanding. arXiv preprint arXiv:2105.00377, 2021. — [arXiv:2105.00377](https://arxiv.org/abs/2105.00377) find
  36. Brenden K Petersen, Mikel Landajuela Larma, Terrell N. Mundhenk, Claudio Prata Santiago, Soo Kyung Kim, and Joanne Taery Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference on Learning Representations, 2021. URL… find
  37. Sergei Popov, Mikhail Lazarev, Vladislav Belavin, Denis Derkach, and Andrey Ustyuzhanin. Symbolic expression generation via variational auto-encoder. PeerJ Computer Science, 9:e1241, 2023. find
  38. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proc…
  39. J. Rapin and O. Teytaud. Nevergrad - A gradient-free optimization platform. https://GitHub.com/FacebookResearch/Nevergrad, 2018. ↗
  40. Samuel H. Rudy, Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Data-driven discovery of partial differential equations. Science Advances, 3(4):e1602614, 2017. doi: 10.1126/ sciadv.1602614. URL https://www.science.org/doi/abs/10.1126/sciadv. 1602614. — [doi:10.1126/sciadv](https://doi.org/10.1126/sciadv) ↗
  41. Cicero dos Santos, Ming Tan, Bing Xiang, and Bowen Zhou. Attentive pooling networks. arXiv preprint arXiv:1602.03609, 2016. find
  42. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=H1gR5iR5FX. ↗
  43. Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. Science Advance, 324(5923):81–85, 2009. ISSN 0036-8075. doi: 10.1126/science.1165893. — [doi:10.1126/science.1165893](https://doi.org/10.1126/science.1165893) find
  44. Parshin Shojaee, Kazem Meidani, Amir Barati Farimani, and Chandan K. Reddy. Transformer-based planning for symbolic regression. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=0rVXQEeFEL. ↗
  45. Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15638–15650, 2022. find
  46. Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji-Rong Wen. A molecular multimodal foundation model associating molecule graphs with natural language. arXiv preprint arXiv:2209.05481, 2022. — [arXiv:2209.05481](https://arxiv.org/abs/2209.05481) find
  47. Kenichi Tamura and Marcus Gallagher. Quantitative measure of nonconvexity for black-box continuous functions. Information Sciences, 476:64–82, 2019. ISSN 0020-0255. doi: https://doi.org/10.1016/j.ins.2018.10.009. URL https://www.sciencedirect.com/science/ article/pii/S0020025518308053. — [doi:10.1016/j.ins.2018.10.009](https://doi.org/10.1016/j.ins.2018.10.009) ↗
  48. Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. Representing numbers in NLP: a survey and a vision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 644–656, Online, June 2021. Associ…
  49. Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay2631, 2020. doi: 10.1126/sciadv.aay2631. URL https://www.science.org/doi/abs/10.1126/sciadv.aay2631. ↗
  50. Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579–2605, 2008. URL http://jmlr.org/papers/v9/ vandermaaten08a.html. ↗
  51. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
  52. Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey. Machine Intelligence Research, pp. 1–36, 2023. find
  53. Sean Welleck, Peter West, Jize Cao, and Yejin Choi. Symbolic brittleness in sequence models: On systematic generalization in symbolic mathematics. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8):8629–8637, Jun. 2022. doi: 10.1609/aaai.v36i8.20841. URL https://ojs.aaai.org/index.… — [doi:10.1609/aaai.v36i8.20841](https://doi.org/10.1609/aaai.v36i8.20841) ↗
  54. Eugene P. Wigner. The unreasonable effectiveness of mathematics in the natural sciences. richard courant lecture in mathematical sciences delivered at new york university, may 11, 1959. Communications on Pure and Applied Mathematics, 13(1):1–14, 1960. doi: https://doi.org/10.1002/ cpa.3160130102. UR… find
  55. Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802, 2023. — [arXiv:2307.10802](https://arxiv.org/abs/2307.10802) find
  56. Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419, 2023. find
  57. Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales. Self-supervised multimodal learning: A survey. arXiv preprint arXiv:2304.01008, 2023. — [arXiv:2304.01008](https://arxiv.org/abs/2304.01008) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/meidani-2023-snip-bridging-mathematical-symbolic-numeric/