Elamur Research
☰
home / papers / 2017 / A simple neural network module for relational reasoning

A simple neural network module for relational reasoning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1706.01427

Summary

Neural networks such as CNNs and MLPs often fail at questions that depend on how entities relate to each other, rather than on the properties of single entities. The authors, from DeepMind, propose the Relation Network (RN), a compact plug-in module that builds relational structure directly into the architecture. An RN applies one shared MLP (g_θ) to every pair of input "objects", sums the resulting pairwise outputs, and passes the sum through a second MLP (f_φ). The design is order-invariant, end-to-end differentiable, and data-efficient because a single relation function is learned for all pairs. Objects can be CNN feature-map cells, LSTM-encoded sentences, or rows of a state-description matrix. RN-augmented models reached state-of-the-art, super-human accuracy on CLEVR. They also solved most bAbI tasks and inferred hidden structure in simulated physical systems. A new diagnostic dataset, Sort-of-CLEVR, shows that strong CNN baselines lack general relational ability but gain it when an RN is attached. The work suggests relational reasoning can be added to deep models with minimal engineering.

Key points

  • Core formula: RN(O) = f_φ(Σ_{i,j} g_θ(o_i, o_j)). Both functions are MLPs, and the summation makes the output independent of object order.
  • CLEVR: The previous best model combined ResNet-101 image features, an LSTM, and stacked attention. It scored 68.5% overall, compared with 92.6% for humans. The RN-augmented model exceeded human performance, and the RN also solved CLEVR when given state descriptions instead of pixels.
  • Sort-of-CLEVR: Each image contains six 2D shapes in 6 colors, with 10 relational and 10 non-relational questions encoded as binary strings. CNN+RN clearly beat CNN+MLP on relational questions, while both performed similarly on non-relational ones.
  • bAbI: The model was trained jointly on all tasks with 10K examples per task. Each support sentence was encoded by a 32-unit LSTM as one object. It passed 18 of the 20 tasks, using a threshold above 95% accuracy.
  • Dynamic physical systems: From ball trajectories over 16 frames, the RN inferred which balls were connected. It also counted connected systems, a task that required learning relations implicitly. The model was also applied to motion capture data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

129 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 119 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1706.01427 · arXiv 1706.01427

@misc{santoro2017simple,
  title = {{A simple neural network module for relational reasoning}},
  author = {Santoro, Adam and Raposo, David and Barrett, David G. T. and Malinowski, Mateusz and Pascanu, Razvan and Battaglia, Peter and Lillicrap, Timothy},
  year = {2017},
  doi = {10.48550/arxiv.1706.01427},
  eprint = {1706.01427},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1706.01427}
}

References

48 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015. find
  2. Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In NIPS, 2016. find
  3. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In EMNLP, 2013. find
  4. Rajarshi Das, Arvind Neelakantan, David Belanger, and Andrew McCallum. Chains of reasoning over entities, relations, and text using recurrent neural networks. arXiv:1607.01426, 2016. find
  5. Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv:1606.01847, 2016. find
  6. Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. Are you talking to a machine? dataset and methods for multilingual image question answering. In NIPS, 2015. find
  7. Marta Garnelo, Kai Arulkumaran, and Murray Shanahan. Towards deep symbolic reinforcement learning. arXiv:1609.05518, 2016. — [arXiv:1609.05518](https://arxiv.org/abs/1609.05518) find
  8. Dave Golland, Percy Liang, and Dan Klein. A game-theoretic approach to generating spatial descriptions. In EMNLP, 2010. find
  9. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 2016. find
  10. Sergio Guadarrama, Lorenzo Riano, Dave Golland, Daniel Gouhring, Yangqing Jia, Dan Klein, Pieter Abbeel, and Trevor Darrell. Grounding spatial relations for human-robot interaction. In IROS, 2013. find
  11. Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335–346, 1990. find
  12. Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. arXiv:1506.05163, 2015. — [arXiv:1506.05163](https://arxiv.org/abs/1506.05163) find
  13. Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. Tracking the world state with recurrent entity networks. In ICLR, 2017. find
  14. Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. arXiv:1704.05526, 2017. find
  15. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. find our card
  16. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. arXiv:1705.03633, 2017. find
  17. Kushal Kafle and Christopher Kanan. arXiv:1703.09684, 2017. — [arXiv:1703.09684](https://arxiv.org/abs/1703.09684) find
  18. Charles Kemp and Joshua B Tenenbaum. The discovery of structural form. Proceedings of the National Academy of Sciences, 105(31):10687–10692, 2008. find
  19. Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv:1609.02907, 2016. find our card
  20. Jayant Krishnamurthy and Thomas Kollar. Jointly learning to parse and perceive: Connecting natural language to the physical world. TACL, 2013. find
  21. Geert-Jan M Kruijff, Hendrik Zender, Patric Jensfelt, and Henrik I Christensen. Situated dialogue and spatial organization: What, where... and why. IJARS, 2007. find
  22. Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. Inducing probabilistic ccg grammars from logical form with higher-order unification. In EMNLP, 2010. find
  23. Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. arXiv:1604.00289, 2016. An analysis of visual question answering algorithms. find
  24. Tian Lan, Weilong Yang, Yang Wang, and Greg Mori. Image retrieval with structured object queries using latent ranking svm. In ECCV, 2012. find
  25. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, 2015. find
  26. Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. ICLR, 2016. find
  27. Percy Liang, Michael I Jordan, and Dan Klein. Learning dependency-based compositional semantics. Computational Linguistics, 2013. find
  28. Gordon D Logan and Daniel D Sadler. A computational analysis of the apprehension of spatial relations. 1996. find
  29. Mateusz Malinowski and Mario Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, 2014. find
  30. Mateusz Malinowski and Mario Fritz. A pooling approach to modelling spatial relations for image retrieval and annotation. arXiv:1411.5190, 2014. — [arXiv:1411.5190](https://arxiv.org/abs/1411.5190) find
  31. Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A deep learning approach to visual question answering. arXiv:1605.02697, 2016. — [arXiv:1605.02697](https://arxiv.org/abs/1605.02697) find
  32. Allen Newell. Physical symbol systems. Cognitive science, 4(2):135–183, 1980. find
  33. Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In ICML, 2016. find
  34. Jack Rae, Jonathan J Hunt, Ivo Danihelka, Timothy Harley, Andrew W Senior, Gregory Wayne, Alex Graves, and Tim Lillicrap. Scaling memory-augmented neural networks with sparse reads and writes. In NIPS, 2016. find
  35. David Raposo, Adam Santoro, David Barrett, Razvan Pascanu, Timothy Lillicrap, and Peter Battaglia. Discovering objects and their relations from entangled scene representations. arXiv:1702.05068, 2017. — [arXiv:1702.05068](https://arxiv.org/abs/1702.05068) find
  36. Mengye Ren, Ryan Kiros, and Richard Zemel. Image question answering: A visual semantic embedding model and a new dataset. In NIPS, 2015. find
  37. Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 2009. find
  38. Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth J Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI, 2011. find
  39. Stefanie Tellex, Thomas Kollar, George Shaw, Nicholas Roy, and Deb Roy. Grounding spatial language for video search. In ICMI, 2010. find
  40. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012. find
  41. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv:1502.05698, 2015. find our card
  42. Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In ICLR, 2015. find
  43. Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In ICML, 2016. find
  44. Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In ECCV, 2016. find
  45. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015. find our card
  46. Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In CVPR, 2016. find
  47. Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. Deep sets. arXiv:1703.06114, 2017. — [arXiv:1703.06114](https://arxiv.org/abs/1703.06114) find
  48. John M Zelle and Raymond J Mooney. Learning to parse database queries using inductive logic programming. In AAAI, 1996. find