Elamur Research⌕
☰
home / papers / 2022 / End-to-end symbolic regression with transformers

End-to-end symbolic regression with transformers

Paper · 2022 · Pierre-Alexandre Kamienny, Stéphane d'Ascoli, Guillaume Lample et al.

Summary

Symbolic regression (SR) looks for a readable mathematical formula that reproduces observed data. Usual pipelines first guess an expression "skeleton" and then fit its constants with BFGS. Genetic programming (GP) repeats this loop many times, so it is accurate but slow. Earlier Transformer methods predicted only skeletons, lagged behind GP and handled at most three input variables. Kamienny et al. instead train a Transformer to output the whole formula, numerical constants included. Its vocabulary combines symbolic tokens for operators and variables with tokens that encode numbers. The model is pre-trained on large synthetic data: random expression trees evaluated on inputs sampled from mixtures of Gaussian and uniform clusters, which are then whitened. The predicted constants can optionally serve as a starting point for BFGS refinement. Ablations show that this end-to-end approach outperforms skeleton prediction. On SRBench, the model approaches leading GP systems such as Operon while running orders of magnitude faster and producing simpler formulas. This makes SR practical where results are needed quickly, for example in real-time settings.

Key points

  • The model works with up to 10 input features (D ≤ 10), compared with 3 in concurrent neural SR work. Numbers are tokenized in base-10 floating-point form with four significant digits.
  • The evaluation used SRBench filtered to 190 datasets: 57 black-box problems, 119 Feynman problems and 14 ODE-Strogatz problems. Each dataset was split 75%/25% into train and test data.
  • On Feynman problems, the model ranks fourth in accuracy (share of problems with R² > 0.99). Its formulas are less complex than those of the top two systems, Operon and SBP-GP.
  • On real-world black-box tasks, it beats classic ML baselines such as XGBoost and Random Forests, with lower inference time and an interpretable output.
  • Without refinement, the end-to-end model is sensitive to label noise, and BFGS refinement restores robustness. Extrapolation stays reasonable even at test-input scale σ = 32. The code is released as a Scikit-learn compatible regressor (github.com/facebookresearch/symbolicregression).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksPhysics and Astronomy · 70 worksStatistical and Nonlinear Physics · 60 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Pierre-Alexandre Kamienny, Stéphane d'Ascoli, Guillaume Lample and François Charton. 2022. End-to-end symbolic regression with transformers. arXiv:2204.10532.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2204.10532 · arXiv 2204.10532

@misc{kamienny2022end,
  title = {{End-to-end symbolic regression with transformers}},
  author = {Pierre-Alexandre Kamienny, Stéphane d'Ascoli, Guillaume Lample and François Charton},
  year = {2022},
  doi = {10.48550/arxiv.2204.10532},
  eprint = {2204.10532},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2204.10532}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Silviu-Marian Udrescu and Max Tegmark. AI Feynman: a Physics-Inspired Method for Symbolic Regression. 2020. arXiv: 1905.11481 [physics.comp-ph].
  2. M. Cranmer et al. ‘Discovering Symbolic Models from Deep Learning with Inductive Biases’. In: ArXiv abs/2006.11287 (2020).
  3. Marta Garnelo, Kai Arulkumaran and Murray Shanahan. ‘Towards deep symbolic reinforcement learning’. In: arXiv preprint arXiv:1609.05518 (2016). — [arXiv:1609.05518](https://arxiv.org/abs/1609.05518) find
  4. Mikel Landajuela et al. ‘Discovering symbolic policies with deep reinforcement learning’. In: International Conference on Machine Learning. PMLR. 2021, pp. 5979–5989. find
  5. Mojtaba Valipour et al. ‘SymbolicGPT: A Generative Transformer Model for Symbolic Regression’. In: arXiv preprint arXiv:2106.14131 (2021). Luca Biggio et al. Neural Symbolic Regression that Scales. 2021. arXiv: 2106.06427 [cs.LG]. — [arXiv:2106.14131](https://arxiv.org/abs/2106.14131) find
  6. William La Cava et al. ‘Contemporary symbolic regression methods and their relative performance’. In: arXiv preprint arXiv:2107.14351 (2021).
  7. Nikos Aréchiga et al. ‘Accelerating Understanding of Scientific Experiments with End to End Symbolic Regression’. In: ArXiv abs/2112.04023 (2021). — [arXiv:2112.04023](https://arxiv.org/abs/2112.04023) find
  8. Silviu-Marian Udrescu and Max Tegmark. ‘Symbolic Pregression: Discovering Physical Laws from Raw Distorted Video’. In: Physical review. E 103 4-1 (2021), p. 043307. find
  9. Anja Butter et al. ‘Back to the Formula – LHC Edition’. In: 2021. Michael Schmidt and Hod Lipson. ‘Age-fitness pareto optimization’. In: Genetic programming theory and practice VIII. Springer, 2011, pp. 129–146. find
  10. Michael Schmidt and Hod Lipson. ‘Distilling free-form natural laws from experimental data’. In: science 324.5923 (2009), pp. 81–85. find
  11. William La Cava et al. ‘Learning concise representations for regression by evolving networks of trees’. In: arXiv preprint arXiv:1807.00981 (2018). Trent McConaghy. ‘FFX: Fast, scalable, deterministic symbolic regression technology’. In: Genetic Programming Theory and Practice IX. Springer, 2011, pp… — [arXiv:1807.00981](https://arxiv.org/abs/1807.00981) find
  12. Marco Virgolin et al. ‘Improving model-based genetic programming for symbolic regression of small expressions’. In: Evolutionary computation 29.2 (2021), pp. 211–237. find
  13. Fabricio Olivetti de França and Guilherme Seidyo Imai Aldeia. ‘Interaction–Transformation Evolutionary Algorithm for Symbolic Regression’. In: Evolutionary computation 29.3 (2021), pp. 367– 390. find
  14. Ignacio Arnaldo, Krzysztof Krawiec and Una-May O’Reilly. ‘Multiple regression genetic programming’. In: Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation. 2014, pp. 879–886. find
  15. Marco Virgolin, Tanja Alderliesten and Peter A. N. Bosman. ‘Linear Scaling with and within Semantic Backpropagation-Based Genetic Programming for Symbolic Regression’. In: Proceedings of the Genetic and Evolutionary Computation Conference. GECCO ’19. Prague, Czech Republic: Association for Computing… — [doi:10.1145/3321707.3321758](https://doi.org/10.1145/3321707.3321758) find
  16. Michael Kommenda et al. ‘Parameter identification for symbolic regression using nonlinear least squares’. In: Genetic Programming and Evolvable Machines 21.3 (2020), pp. 471–501. doi: 10.1007/s10710-019-09371-3. url: https://doi.org/10.1007/s10710-019-09371-3. — [doi:10.1007/s10710-019-09371-3](https://doi.org/10.1007/s10710-019-09371-3) ↗
  17. Brenden K Petersen et al. ‘Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients’. In: arXiv preprint arXiv:1912.04871 (2019). Georg Martius and Christoph H Lampert. ‘Extrapolation and learning equations’. In: arXiv preprint arXiv:1610.02995 (2016)… — [arXiv:1912.04871](https://arxiv.org/abs/1912.04871) find
  18. Subham Sahoo, Christoph Lampert and Georg Martius. ‘Learning equations for extrapolation and control’. In: International Conference on Machine Learning. PMLR. 2018, pp. 4442–4450. Ashish Vaswani et al. ‘Attention is all you need’. In: Advances in neural information processing systems. 2017, pp. 5998…
  19. Guillaume Lample and François Charton. ‘Deep learning for symbolic mathematics’. In: arXiv preprint arXiv:1912.01412 (2019).
  20. François Charton, Amaury Hayat and Guillaume Lample. ‘Learning advanced mathematical computations from examples’. In: arXiv preprint arXiv:2006.06462 (2020). François Charton. ‘Linear algebra with transformers’. In: arXiv preprint arXiv:2112.01898 (2021).
  21. Christopher Hahn et al. ‘Teaching temporal logics to neural networks’. In: arXiv preprint arXiv:2003.04218 (2020).
  22. Stanislas Polu and Ilya Sutskever. ‘Generative language modeling for automated theorem proving’. In: arXiv preprint arXiv:2009.03393 (2020). Stéphane d’Ascoli et al. ‘Deep Symbolic Regression for Recurrent Sequences’. In: arXiv preprint arXiv:2201.04600 (2022).
  23. Roger Guimerà et al. ‘A Bayesian machine scientist to aid in the solution of challenging scientific problems’. In: Science advances 6.5 (2020), eaav6971. Patrick Kidger. SympyTorch. https://github.com/patrick-kidger/sympytorch. 2021. ↗
  24. Richard Zou Horace He. functorch: JAX-like composable function transforms for PyTorch. https://github.com/pytorch/functorch. 2021. ↗
  25. Sam Wiseman and Alexander M. Rush. Sequence-to-Sequence Learning as Beam-Search Optimization. 2016. doi: 10.48550/ARXIV.1606.02960. url: https://arxiv.org/abs/1606.02960. ↗
  26. Jerome H Friedman. ‘Greedy function approximation: a gradient boosting machine’. In: Annals of statistics (2001), pp. 1189–1232. find
  27. Steven H. Strogatz. Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry and Engineering. Westview Press, 2000. find
  28. Ying Jin et al. Bayesian Symbolic Regression. 2020. arXiv: 1910.08892 [stat.ME]. — [arXiv:1910.08892](https://arxiv.org/abs/1910.08892) find
  29. T. Nathan Mundhenk et al. Symbolic Regression via Neural-Guided Genetic Programming Population Seeding. 2021. arXiv: 2111.00053 [cs.NE]. — [arXiv:2111.00053](https://arxiv.org/abs/2111.00053) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kamienny-2022-end-end-symbolic-regression-transformers/