Elamur Research⌕
☰
home / papers / 2018 / Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding

Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding

Paper · 2018 · Kexin Yi, Jiajun Wu, Chuang Gan et al.

Summary

End-to-end visual question answering (VQA) models often struggle with compositional, multi-step reasoning. They also reveal little about how they arrive at an answer. Yi et al. propose NS-VQA, which separates perception and language understanding from reasoning. A neural scene parser turns an image into a compact table of objects and their attributes. It uses Mask R-CNN proposals plus a ResNet-34 that estimates 3D position and pose. An attention-based seq2seq LSTM translates the question into a hierarchical program of functional modules. This parser is first trained on a few annotated programs and then refined with REINFORCE. A deterministic symbolic executor runs the program over the object table to produce the answer. On CLEVR, the system reaches 99.8% accuracy with only 270 annotated programs and 4K images. It beats the IEP baseline on CLEVR-Humans under limited program supervision and transfers to a new Minecraft-based dataset. The work shows that building symbolic structure into a neural pipeline can raise accuracy, reduce annotation and memory needs, and make every reasoning step inspectable.

Key points

  • Pipeline: Mask R-CNN segments objects and predicts color, material, size and shape, dropping proposals scored below 0.9. A ResNet-34 receives each segment together with the full image at 224×224 and predicts spatial attributes.
  • CLEVR: 99.8% accuracy using 270 program annotations and 4K training images. The structural scene representation cuts memory cost for offline QA by about 99% compared with state-of-the-art methods.
  • CLEVR-Humans: the question parser is pretrained on CLEVR programs, with fixed GloVe embeddings. It is then fine-tuned with REINFORCE for up to 1M iterations with early stopping, and it clearly outperforms IEP when few programs are annotated.
  • Minecraft extension: 10,000 scenes were rendered via Malmo, each with 3–6 objects drawn from 12 entity types and encoded as 18-dim vectors. The split is 9,000 images / 88,109 questions for training and 1,000 / 9,761 for testing. Accuracy is 71.1%, 72.4%, 86.9% and 87.3% with 50, 100, 200 and 500 annotated programs.
  • Most Minecraft errors come from heavily occluded objects rather than from question parsing. The authors name generalizing structured representations to truly novel situations as an open challenge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations🏢IBMArmonk · United States154 works · 72,606 citations🎓Harvard UniversityCambridge · United States126 works · 45,130 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States76 works · 32,207 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksVision & language · 811 works

Cite

Cite as Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli and Joshua B. Tenenbaum. 2018. Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. arXiv:1810.02338.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1810.02338 · arXiv 1810.02338

@misc{yi2018neural,
  title = {{Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding}},
  author = {Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli and Joshua B. Tenenbaum},
  year = {2018},
  doi = {10.48550/arxiv.1810.02338},
  eprint = {1810.02338},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1810.02338}
}

References

50 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Somak Aditya, Yezhou Yang, and Chitta Baral. Explicit reasoning over end-to-end neural architectures for visual question answering. In AAAI, 2018. 3 find
  2. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. In NAACL-HLT, 2016. 3
  3. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015. 1, 3 find
  4. Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention. In ICLR, 2015. 2 find
  5. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. 4
  6. Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. In ICLR, 2017. 2 find
  7. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In EMNLP, 2013. 2
  8. Yonatan Bisk, Kevin J Shih, Yejin Choi, and Daniel Marcu. Learning interpretable spatial operations in a rich 3d blocks world. In AAAI, 2018. 3 find
  9. Qingxing Cao, Xiaodan Liang, Bailing Li, Guanbin Li, and Liang Lin. Visual question reasoning on general dependency tree. In CVPR, 2018. 3, 5 find
  10. SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In NIPS, 2016. 2 find
  11. Chuang Gan, Yandong Li, Haoxiang Li, Chen Sun, and Boqing Gong. Vqs: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation. In ICCV, 2017. 3 find
  12. Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He. Detectron. https://github.com/facebookresearch/detectron, 2018. 5 ↗
  13. Omer Goldman, Veronica Latcinnik, Udi Naveh, Amir Globerson, and Jonathan Berant. Weakly-supervised semantic parsing with abstract examples. In ACL, 2018. 3 find
  14. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. 3
  15. Kelvin Guu, Panupong Pasupat, Evan Zheran Liu, and Percy Liang. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In ACL, 2017. 3
  16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2015. 4
  17. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017. 4, 5 find
  18. Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: learning abstract hierarchical compositional visual concepts. In ICLR, 2018. 2 find
  19. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8):1735–1780, 1997. 4 find
  20. Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In CVPR, 2017. 3, 5 find
  21. Drew A Hudson and Christopher D Manning. Compositional attention networks for machine reasoning. In ICLR, 2018. 3, 5, 7
  22. Allan Jabri, Armand Joulin, and Laurens van der Maaten. Revisiting visual question answering baselines. In ECCV, 2016. 3 find
  23. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017a. 1, 3, 6, 7
  24. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In ICCV, 2017b. 1, 3, 4, 5, 6, 7, 8
  25. Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In IJCAI, 2016. 8 find
  26. Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Joshua B Tenenbaum. Deep convolutional inverse graphics network. In NIPS, 2015. 2 find
  27. Percy Liang, Michael I Jordan, and Dan Klein. Learning dependency-based compositional semantics. Computational Linguistics, 39(2):389–446, 2013. 2 find
  28. Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017. 5
  29. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In NIPS, 2016. 3 find
  30. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. In EMNLP, 2015. 4
  31. M Malinowski and M Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, 2014. 1, 3 find
  32. David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar. Transparency by design: Closing the gap between performance and interpretability in visual reasoning. In CVPR, 2018. 3, 5, 7 find
  33. Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta, and Laurens van der Maaten. Learning by asking questions. In CVPR, 2018. 3 find
  34. Arvind Neelakantan, Quoc V Le, and Ilya Sutskever. Neural programmer: Inducing latent programs with gradient descent. In ICLR, 2016. 2 find
  35. Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. Neuro-symbolic program synthesis. In ICLR, 2017. 2
  36. Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014. 8
  37. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018. 3, 5, 7
  38. Anselm Rothe, Brenden M Lake, and Todd Gureckis. Question asking as program generation. In NIPS, 2017. 3 find
  39. Adam Santoro, David Raposo, David GT Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In NIPS, 2017. 3, 5
  40. N Siddharth, T. B. Paige, J.W. Meent, A. Desmaison, N. Goodman, P. Kohli, F. Wood, and P. Torr. Learning disentangled representations with semi-supervised deep generative models. In NIPS, 2017. 2 find
  41. Joseph Suarez, Justin Johnson, and Fei-Fei Li. Ddrprog: A clevr differentiable dynamic reasoning programmer. arXiv:1803.11361, 2018. 3, 5 — [arXiv:1803.11361](https://arxiv.org/abs/1803.11361) find
  42. Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imagination. In ICLR, 2018. 2 find
  43. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. Grammar as a foreign language. In NIPS, 2015. 2 find
  44. Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick. Explicit knowledge-based reasoning for visual question answering. In IJCAI, 2017. 3 find
  45. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. MLJ, 8(3-4):229–256, 1992. 5 find
  46. Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In CVPR, 2017. 2, 8 find
  47. Jimei Yang, Scott E Reed, Ming-Hsuan Yang, and Honglak Lee. Weakly-supervised disentangling with recurrent transformations for 3d view synthesis. In NIPS, 2015. 2 find
  48. Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In CVPR, 2016. 1, 3
  49. Alan Yuille and Daniel Kersten. Vision as bayesian inference: analysis by synthesis? Trends Cogn. Sci., 10(7): 301–308, 2006. 2 find
  50. Chen Zhu, Yanpeng Zhao, Shuaiyi Huang, Kewei Tu, and Yi Ma. Structured attentions for visual question answering. In ICCV, 2017. 3 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yi-2018-neural-symbolic-vqa-disentangling-reasoning/