Elamur Research⌕
☰
home / papers / 2022 / What is my math transformer doing? — Three results on interpretability and generalization

What is my math transformer doing? — Three results on interpretability and generalization

Paper · 2022 · François Charton

Summary

Transformers used for mathematics face three common criticisms: they are opaque, they can produce nonsensical answers, and they fail outside their training distribution. Charton tests these claims on symmetric-matrix linear algebra tasks: eigenvalue computation, diagonalization and inversion. He uses sequence-to-sequence transformers trained only from examples, with no task-specific inductive bias, following his earlier setup.

On 50,000 random test matrices, even wrong outputs keep core theoretical structure. Eigenvalues come out nearly exact, and the predicted eigenvector matrices have unit-norm rows and columns. Most diagonalization errors come from imperfect orthogonality. Inversion failures occur mainly on ill-conditioned inputs. This means simple condition-number statistics can serve as verifiers that anticipate failure.

He also varies the eigenvalue distribution of the training matrices. The usual Wigner (semicircle) ensemble generalizes poorly. Gaussian or Laplace eigenvalue distributions transfer to all seven test distributions, need fewer examples, and scale to larger matrices. Test matrices with only positive eigenvalues are very rare in such training data, so these results argue against pure interpolation from memorized examples. They also show that training-data design is a practical lever for robustness.

Key points

  • Diagonalization (5×5): accuracy is 92.0%.
    • Eigenvalues fall within 1% relative error in 99.9% of cases.
    • All rows and columns of H have near-unit norm in 98.9% of cases.
    • These properties persist in weaker models: a half-trained model (70% accuracy) and a 6×6 model (43% accuracy).
  • Diagonalization verifier: a threshold of 1.045 on the condition number of the predicted H separates successes from failures in 99.3% of test cases.
  • Inversion (5×5): accuracy is 89.0%.
    • Predictions lie within 5% of the true inverse in 98.2% of cases.
    • The rule c(M) < 62, computed on the input alone, predicts the outcome 98.0% of the time.
  • Out-of-distribution behavior:
    • Models trained on Gaussian or Laplace eigenvalue distributions generalize broadly.
    • Models trained only on positive matrices fail on matrices with negative eigenvalues.
    • Training distributions with bounded support transfer worse than unbounded ones.
  • Data efficiency:
    • For 8×8 eigenvalues, Gaussian and Laplace models reach 99% on a semicircle test set after 11.4M and 13.2M examples. The semicircle model is at 0.5% after 36M.
    • With deeper encoders, 10×10 Wigner eigenvalues reach 100% accuracy. The earlier best result was 25% after 360M examples.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputational Theory and Mathematics · 496 works

🧭 Topics 1Research subjects and languages the work is about.Interpretability · 156 works

Cite

Cite as François Charton. 2022. What is my math transformer doing? — Three results on interpretability and generalization. arXiv:2211.00170.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2211.00170 · arXiv 2211.00170

@misc{charton2022what,
  title = {{What is my math transformer doing? — Three results on interpretability and generalization}},
  author = {François Charton},
  year = {2022},
  doi = {10.48550/arxiv.2211.00170},
  eprint = {2211.00170},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2211.00170}
}

References

13 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascandolo. Neural symbolic regression that scales. arXiv preprint arXiv:2106.06427, 2021. — [arXiv:2106.06427](https://arxiv.org/abs/2106.06427) find
  2. François Charton. Linear algebra with transformers. arXiv preprint arXiv:2112.01898, 2021.
  3. Aurélien Dersy, Matthew D. Schwartz, and Xiaoyuan Zhang. Simplifying polylogarithms with machine learning. arXiv preprint arXiv:2206.04115, 2022. — [arXiv:2206.04115](https://arxiv.org/abs/2206.04115) find
  4. Gene H. Golub and Charles F. van Loan. Matrix Computations. JHU Press, fourth edition, 2013. find
  5. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  6. Guillaume Lample and François Charton. Deep learning for symbolic mathematics. arXiv preprint arXiv:1912.01412, 2019.
  7. Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. find
  8. Madan Lal Mehta. Random Matrices. Academic Press, 3rd edition, 2004. find
  9. Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Investigating the limitations of transformers with simple arithmetic tasks. arXiv preprint arXiv:2102.13019, 2021.
  10. Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.03393, 2020.
  11. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010, 2017.
  12. Sean Welleck, Peter West, Jize Cao, and Yejin Choi. Symbolic brittleness in sequence models: on systematic generalization in symbolic mathematics. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8629–8637, 2022. find
  13. Gal Yehuda, Moshe Gabel, and Assaf Schuster. It’s not what machines can learn, it’s what we cannot teach. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10831–10841. PMLR,… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/charton-2022-my-math-transformer-doing-three/