Elamur Research⌕
☰
home / papers / 2018 / Measuring abstract reasoning in neural networks

Measuring abstract reasoning in neural networks

Paper · 2018 · David G. T. Barrett, Felix Hill, Adam Santoro et al.

Summary

The authors from DeepMind examine whether neural networks truly learn to reason abstractly or merely exploit superficial statistical regularities in the data. Taking the Raven's Progressive Matrices IQ test as a model, they built a puzzle generator and a large open dataset, PGM (Procedurally Generated Matrices). Each task is a 3×3 matrix of images with one cell missing, which must be chosen from several candidates. The abstract structure of each problem is specified explicitly, so the training and test sets can be separated in a controlled way, through so-called generalization regimes. Purely convolutional models, including ResNet-50, performed poorly. The proposed Relation Network variant called WReN, designed to compare parts of the image with one another, performs considerably better. It interpolates attribute values well and applies known rules in novel combinations. At the same time, the model weakens markedly when it must extrapolate beyond the values it has seen or deal with unfamiliar attributes. Generalization improves noticeably if the model is additionally trained to predict a symbolic explanation of its answer. The work offers a tool for honestly measuring the abstract reasoning of models and for developing it.

Key points

  • The structure of each PGM is a set of 1–4 triples [relation, object, attribute]. Relations: progression, XOR, OR, AND, consistent union. Objects: shape, line. Attributes: size, type, colour, position, number. Separately, there is a regime with distracting attributes that vary randomly.
  • Several generalization regimes are studied: neutral, interpolation, extrapolation, held-out attribute (shape-colour, line-type), held-out triples and held-out pairs of triples.
  • There are 29 possible unique triples, 7 of which are held out for testing only. Of the 400 permissible pairs of triples, 360 are assigned to training and 40 to testing.
  • WReN substantially outperforms CNN and ResNet-50. However, the model does not transfer a known relation to a different object type, whereas for a human this would be trivial.
  • An auxiliary task of predicting a symbolic explanation of the answer (meta-targets) improves both accuracy and generalization ability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,462 citations🎓University of WashingtonSeattle · United States402 works · 212,762 citations🎓University of TorontoToronto · Canada110 works · 179,061 citations🏢Meta (Israel)Tel Aviv · Israel46 works · 42,810 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as David G. T. Barrett, Felix Hill, Adam Santoro, Ari S. Morcos and Timothy Lillicrap. 2018. Measuring abstract reasoning in neural networks. arXiv:1807.04225.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1807.04225 · arXiv 1807.04225

@misc{barrett2018measuring,
  title = {{Measuring abstract reasoning in neural networks}},
  author = {David G. T. Barrett, Felix Hill, Adam Santoro, Ari S. Morcos and Timothy Lillicrap},
  year = {2018},
  doi = {10.48550/arxiv.1807.04225},
  eprint = {1807.04225},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1807.04225}
}

References

33 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andreas, J., Klein, D., and Levine, S. Modular multitask reinforcement learning with policy sketches. arXiv preprint arXiv:1611.01796, 2016. find
  2. Botvinick, M., Barrett, D., Battaglia, P., de Freitas, N., Kumaran, D., Leibo, J., Lillicrap, T., Modayil, J., Mohamed, S., Rabinowitz, N., et al. Building machines that learn and think for themselves: Commentary on lake et al., behavioral and brain sciences, 2017. arXiv preprint arXiv:1711.08378, 2… — [arXiv:1711.08378](https://arxiv.org/abs/1711.08378) find
  3. Carpenter, P. A., Just, M. A., and Shell, P. What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test. Psychological review, 97(3):404, 1990. find
  4. Clark, P. and Etzioni, O. My computer is an honor student-but how intelligent is it? standardized tests as a measure of ai. AI Magazine, 37(1):5–12, 2016. find
  5. Davis, E. The limitations of standardized science tests as benchmarks for artificial intelligence research: Position paper. arXiv preprint arXiv:1411.1629, 2014. — [arXiv:1411.1629](https://arxiv.org/abs/1411.1629) find
  6. Fleuret, F., Li, T., Dubout, C., Wampler, E. K., Yantis, S., and Geman, D. Comparing machines and humans on a visual categorization test. Proceedings of the National Academy of Sciences, 108(43):17621–17625, 2011. find
  7. Flynn, J. R. Massive iq gains in 14 nations: What iq tests really measure. Psychological bulletin, 101(2):171, 1987. find
  8. Garnelo, M., Arulkumaran, K., and Shanahan, M. Towards deep symbolic reinforcement learning. arXiv preprint arXiv:1609.05518, 2016. — [arXiv:1609.05518](https://arxiv.org/abs/1609.05518) find
  9. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  10. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016. find
  11. Higgins, I., Sonnerat, N., Matthey, L., Pal, A., Burgess, C. P., Botvinick, M., Hassabis, D., and Lerchner, A. Scan: learning abstract hierarchical compositional visual concepts. arXiv preprint arXiv:1707.03389, 2017. — [arXiv:1707.03389](https://arxiv.org/abs/1707.03389) find
  12. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Comput., 9(8):1735–1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. find
  13. Hoshen, D. and Werman, M. Iq of neural networks. arXiv preprint arXiv:1710.01692, 2017. — [arXiv:1710.01692](https://arxiv.org/abs/1710.01692) find
  14. Jaeggi, S. M., Buschkuehl, M., Jonides, J., and Perrig, W. J. Improving fluid intelligence with training on working memory. Proceedings of the National Academy of Sciences, 105(19):6829–6833, 2008. find
  15. Jo, J. and Bengio, Y. Measuring the tendency of cnns to learn surface statistical regularities. arXiv preprint arXiv:1711.11561, 2017. — [arXiv:1711.11561](https://arxiv.org/abs/1711.11561) find
  16. Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., and Girshick, R. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pp. 1988–1997.
  17. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  18. Lake, B. M. and Baroni, M. Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks. arXiv preprint arXiv:1711.00350, 2017. find
  19. LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 521(7553):436, 2015. find
  20. Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146, 2017. find
  21. Lovett, A. and Forbus, K. Modeling visual problem solving as analogical reasoning. Psychological review, 124(1):60, 2017. find
  22. Marcus, G. Deep learning: A critical appraisal. arXiv preprint arXiv:1801.00631, 2018.
  23. Marcus, G., Rossi, F., and Veloso, M. Beyond the turing test. Ai Magazine, 37(1):3–4, 2016. find
  24. Ramakrishnan, S. K., Pal, A., Sharma, G., and Mittal, A. An empirical evaluation of visual question answering for novel objects. arXiv preprint arXiv:1704.02516, 2017. — [arXiv:1704.02516](https://arxiv.org/abs/1704.02516) find
  25. Raposo, D., Santoro, A., Barrett, D., Pascanu, R., Lillicrap, T., and Battaglia, P. Discovering objects and their relations from entangled scene representations. 2017. find
  26. Raven, J. C. et al. Raven’s progressive matrices. Western Psychological Services, 1938. find
  27. Santoro, A., Raposo, D., Barrett, D. G., Malinowski, M., Pascanu, R., Battaglia, P., and Lillicrap, T. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pp. 4974–4983, 2017.
  28. Snow, R. E., Kyllonen, P. C., and Marshalek, B. The topography of ability and learning correlations. Advances in the psychology of human intelligence, 2(S 47):103, 1984. find
  29. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
  30. Te Nijenhuis, J., Voskuijl, O. F., and Schijve, N. B. Practice and coaching on iq tests: Quite a lot of g. International Journal of Selection and Assessment, 9(4):302–308, 2001. find
  31. Wang, K. and Su, Z. Automatic generation of ravens progressive matrices. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015. find
  32. Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014.
  33. Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/barrett-2018-measuring-abstract-reasoning-neural-networks/