Elamur Research⌕
☰
home / papers / 2018 / Compositional Attention Networks for Machine Reasoning

Compositional Attention Networks for Machine Reasoning

Paper · 2018 · Drew A. Hudson and Christopher D. Manning

Summary

Neural networks often struggle with compositional, multi-step reasoning. They tend to latch onto statistical shortcuts, and the modular alternatives depend on handcrafted programs, parsers or complex reinforcement learning training. Hudson and Manning introduce the MAC network, a fully differentiable architecture trained end-to-end with ordinary backpropagation. It tackles a question by chaining recurrent Memory, Attention and Composition cells, each carrying out one reasoning step. Every cell keeps two separate states: a control state that attends to parts of the question, and a memory state that accumulates evidence read from the image. Question and image interact only indirectly, through attention distributions and gating, rather than being merged into a shared vector space. On the CLEVR visual question answering benchmark, MAC sets a new state of the art and cuts the previous best error rate in half. It also does well on human-written questions and needs far less training data. The work shows that strong structural priors can produce interpretable, data-efficient reasoning without strong supervision.

Key points

  • MAC reaches 98.9% accuracy on CLEVR, making only half as many mistakes as the strongest earlier model. It is especially strong on counting and aggregation questions.
  • MAC reaches strong results with roughly 5x less data than existing models. It also generalizes to CLEVR-Human questions involving new concepts such as obstructions and superlatives.
  • Each cell has a control unit, a read unit and a write unit. The control unit applies soft attention over the question words, the read unit attends over the image regions, and the write unit updates the memory.
  • The inputs are encoded as follows:
    • Questions pass through a biLSTM.
    • Images pass through ImageNet-pretrained ResNet101 conv4 features and two CNN layers, giving a 14×14 knowledge base.
  • A TensorFlow implementation is released at github.com/stanfordnlp/mac-network.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Drew A. Hudson and Christopher D. Manning. 2018. Compositional Attention Networks for Machine Reasoning. arXiv:1803.03067.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1803.03067 · arXiv 1803.03067

@misc{hudson2018compositional,
  title = {{Compositional Attention Networks for Machine Reasoning}},
  author = {Drew A. Hudson and Christopher D. Manning},
  year = {2018},
  doi = {10.48550/arxiv.1803.03067},
  eprint = {1803.03067},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1803.03067}
}

References

29 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1955–1960, 2016.
  2. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 39–48, 2016a.
  3. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. In Proceedings of NAACL-HLT, pp. 1545–1554, 2016b.
  4. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2425–2433, 2015. find
  5. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. 2015.
  7. Léon Bottou. From machine learning to machine reasoning. Machine learning, 94(2):133–149, 2014. find
  8. Prithvijit Chattopadhyay, Ramakrishna Vedantam, Ramprasaath R Selvaraju, Dhruv Batra, and Devi Parikh. Counting everyday objects in everyday scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1135–1144, 2017. find
  9. Marta Garnelo, Kai Arulkumaran, and Murray Shanahan. Towards deep symbolic reinforcement learning. arXiv preprint arXiv:1609.05518, 2016. — [arXiv:1609.05518](https://arxiv.org/abs/1609.05518) find
  10. Alex Graves, Greg Wayne, and Ivo Danihelka. arXiv:1410.5401, 2014. find
  11. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538 (7626):471–476, 2016. find
  12. Akshay Kumar Gupta. Survey of visual question answering: Datasets and techniques. arXiv preprint arXiv:1705.03865, 2017. — [arXiv:1705.03865](https://arxiv.org/abs/1705.03865) find
  13. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  14. Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 804–813, 2017. find
  15. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448–456, 2015. find
  16. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pp. 1988–1997.
  17. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2989–2998, 2017b.
  18. Neural turing machines. arXiv preprint Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  19. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. In International Conference on Machine Learning, pp. 1378–1387, 2016.
  20. Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2017. find
  21. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In Advances In Neural Information Processing Systems, pp. 289– 297, 2016. find
  22. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1400–1409, 2016. find
  23. Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. arXiv preprint arXiv:1709.07871, 2017.
  24. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015.
  25. Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Tim Lillicrap. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pp. 4974–4983, 2017.
  26. Bob L Sturm. A simple method to determine if a music information retrieval system is a horse. IEEE Transactions on Multimedia, 16(6):1636–1644, 2014. find
  27. Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In International Conference on Machine Learning, pp. 2397–2406, 2016. find
  28. Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 21–29, 2016.
  29. Network Dimension Control Unit 0.9 0.9 standard control unit 0.8 0.99 Accuracy (Val) Accuracy (Val) Accuracy (Val) attention over word vectors 0.8 0.7 w/o word attention (c2) w/o control 0.6 0.98 128d 0.7 256d 0.5 512d 0.4 0.6 0.97 Epoch Epoch Figure 15: Learning curves for ablated MAC variants (acc… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hudson-2018-compositional-attention-networks-machine-reasoning/