Elamur Research⌕
☰
home / papers / 2024 / Machine learning and information theory concepts towards an AI Mathematician

Machine learning and information theory concepts towards an AI Mathematician

Paper · 2024 · Yoshua Bengio and Nikolay Malkin

Summary

This position essay asks why modern AI handles language well but still struggles with mathematical reasoning, and what objective could drive mathematical discovery rather than just proving given theorems. Using the psychological split between intuitive "system 1" and deliberate "system 2" thinking, the authors argue that deep learning mostly covers the intuitive side. They link reasoning to inductive biases such as a narrow working-memory bottleneck, which pushes the mind to compress information into abstractions. Drawing on learning theory, Bayesian inference and Occam's Razor, they treat compression as a general principle of generalization and apply it to mathematics. Their central hypothesis is that a good collection of theorems is short to describe yet lies only a few derivation steps from many provable statements. They sketch a system that generates conjectures under this objective and tries to prove them with goal-conditioned reinforcement learning, using lemmas as subgoals. The paper has no experiments or datasets. Its value is a principled, information-theoretic target for building an AI mathematician that could also inform AI for science.

Key points

  • The work is a conceptual essay (arXiv:2403.04571, March 2024) and reports no empirical results. Its focus is conjecture discovery, not proving fixed theorems.
  • Under Bayesian reasoning, a theory that needs n fewer bits gets a 2^n prior advantage when both explain the data equally well. The authors use this to motivate compression as the criterion for interesting mathematics.
  • They cite Miller's "7 ± 2" working-memory limit and neuroscience comparisons of monkeys and humans by Dehaene et al. These support the idea of the mind as a compression bottleneck that produces discrete, compositional thoughts.
  • Naming lemmas, definitions and tactics counts as compression: a symbol that is reused shortens the overall body of proofs. This connects theorem proving to program synthesis through the Curry-Howard correspondence.
  • The proposed recipe combines pre-training on a corpus like mathlib with AlphaGo-style self-exploration. Theorem usefulness would be estimated through stochastic-gradient proxies, such as how often a theorem gets used.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de MontréalMontreal · Canada96 works · 92,262 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada73 works · 14,704 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputational Theory and Mathematics · 496 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Yoshua Bengio and Nikolay Malkin. 2024. Machine learning and information theory concepts towards an AI Mathematician. arXiv:2403.04571.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.04571 · arXiv 2403.04571

@misc{bengio2024machine,
  title = {{Machine learning and information theory concepts towards an AI Mathematician}},
  author = {Yoshua Bengio and Nikolay Malkin},
  year = {2024},
  doi = {10.48550/arxiv.2403.04571},
  eprint = {2403.04571},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.04571}
}

References

15 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Susan Amin, Maziar Gomrokchi, Harsh Satija, Herke van Hoof, and Doina Precup. A survey of exploration methods in reinforcement learning. arXiv preprint arXiv:2109.00157, 2021. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. I… — [arXiv:2109.00157](https://arxiv.org/abs/2109.00157) find
  2. Neural Information Processing Systems, 2001. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. International Conference on Machine Learning, 2009. Yoshua Bengio, Yann Lecun, and Geoffrey Hinton. Deep learning for ai. Communications of the ACM, 64(7):58–65, 20… find
  3. Springer, 2013. Léon Bottou. Large-scale machine learning with stochastic gradient descent. International Conference on Computational Statistics, 2010. David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine learning, 15:201–221, 1994. Maxwell Crouse, Ibrah… find
  4. Programming Language Design and Implementation, 2021. Herbert Gelernter. Realization of a geometry theorem-proving machine. International Conference on Information Processing, 1959. Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition. Proceedings of the Roya…
  5. Open Court Publishing, 2006. Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110:457–506, 2021. find
  6. Xu Ji, Eric Elmoznino, George Deane, Axel Constant, Guillaume Dumas, Guillaume Lajoie, Jonathan Simon, and Yoshua Bengio. Sources of richness and ineffability for phenomenally conscious states. arXiv preprint arXiv:2302.06403, 2023. — [arXiv:2302.06403](https://arxiv.org/abs/2302.06403) find
  7. Daniel Kahneman. Thinking, Fast and Slow. Macmillan, 2011. Junsu Kim, Younggyo Seo, and Jinwoo Shin. Landmark-guided subgoal generation in hierarchical reinforcement learning. find
  8. Neural Information Processing Systems, 2021. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, 2015. David JC MacKay. Information theory, inference and learning algorithms. find
  9. Cambridge University Press, 2003. The mathlib community. The Lean mathematical library. International Conference on Certified Programs and Proofs, 2020. George A Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2):8…
  10. Springer, 1990. Stuart Russell. Human compatible: Artificial intelligence and the problem of control. find
  11. Penguin, 2019. Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234–242, 1992. Daniel Selsam, Leonardo de Moura, Kevin Buzzard, Reid Barton, Percy Liang, Sarah Loos, and Freek Wiedijk. The IMO Grand Challenge. URL https://… find
  12. Springer, 1982. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. find
  13. Neural Information Processing Systems, 2017. Jiřı́ Vyskočil, David Stanovský, and Josef Urban. Automated proof compression by invention of new definitions. In Edmund M. Clarke and Andrei Voronkov, editors, Logic for Programming, Artificial Intelligence, and Reasoning, pages 447–462. find
  14. Springer, 2010. Philip Wadler. Propositions as types. Communications of the ACM, 58(12):75–84, 2015. find
  15. Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. LeanDojo: Theorem proving with retrieval-augmented language models. Neural Information Processing Systems, 2023. Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. MiniF2F:…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bengio-2024-machine-learning-information-theory-concepts/