Elamur Research⌕
☰
home / papers / 2019 / The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

Paper · 2019 · Jiayuan Mao, Chuang Gan, Pushmeet Kohli et al.

Summary

Visual reasoning systems that run explicit programs are usually trained with costly extra annotations, such as ground-truth programs or labelled object attributes. This paper presents the Neuro-Symbolic Concept Learner (NS-CL), which needs only images paired with questions and answers. NS-CL has three parts. A perception module builds a latent vector for each detected object. A semantic parser turns each question into a program in a domain-specific language. A quasi-symbolic executor runs that program over the object representations. The executor is deterministic but differentiable, so the visual features and concept embeddings learn from answer correctness. The parser, whose output is discrete, is trained with REINFORCE. Training follows a curriculum that moves from small scenes and short questions to crowded scenes and compositional queries. NS-CL reaches state-of-the-art accuracy on CLEVR without any program annotations. It also generalizes to larger scenes, longer programs, new attribute combinations (CLEVR-CoGenT) and new concepts, and it transfers to image-caption retrieval. Because every reasoning step is explicit, errors can be traced and ambiguous questions detected. This makes the approach a useful template for interpretable, weakly supervised grounding of language.

Key points

  • Architecture: A pretrained Mask R-CNN proposes objects, and ResNet-based features represent each one. Attributes such as Shape and Color act as neural operators that map objects into a joint visual-semantic space, where classification uses similarity to concept embeddings.
  • Supervision compared with prior models: IEP and TbD rely on 700K program annotations, and NS-VQA uses 0.2K programs plus attribute labels. NS-CL uses zero programs and no attribute labels.
  • Curriculum: Training starts with scenes of at most 3 objects and simple attribute questions. It then adds relational questions (e.g., "right of", same material) and finally complex, highly compositional queries.
  • Evaluation: Experiments cover CLEVR, CLEVR-CoGenT, a Minecraft reasoning dataset and natural images from VQS. On VQS, NS-CL performs comparably to an MLP baseline and the MAC network.
  • Interpretability: Execution traces show where reasoning fails, for example a misread material or a missed detection. The executor also flags references that are ambiguous or point to objects that do not exist.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,462 citations🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum and Jiajun Wu. 2019. The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. arXiv:1904.12584.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.12584 · arXiv 1904.12584

@misc{mao2019neuro,
  title = {{The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision}},
  author = {Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum and Jiajun Wu},
  year = {2019},
  doi = {10.48550/arxiv.1904.12584},
  eprint = {1904.12584},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.12584}
}

References

52 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Omri Abend, Tom Kwiatkowski, Nathaniel J Smith, Sharon Goldwater, and Mark Steedman. Bootstrapping language acquisition. Cognition, 2017. find
  2. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018. find
  3. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. In NAACL-HLT, 2016.
  4. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In ICCV, 2015. find
  5. Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. TACL, 2013.
  6. Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori. Object level visual reasoning in videos. In ECCV, 2018. find
  7. Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, and Ram Nevatia. Abc-cnn: An attention based convolutional neural network for visual question answering. arXiv:1511.05960, 2015. — [arXiv:1511.05960](https://arxiv.org/abs/1511.05960) find
  8. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014. — Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
  9. Grzegorz Chrupała, Akos Kádár, and Afra Alishahi. Learning language through pictures. In ACL, 2015. find
  10. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. find
  11. Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015.
  12. Li Dong and Mirella Lapata. Language to logical form with neural attention. In ACL, 2016.
  13. Afsaneh Fazly, Afra Alishahi, and Suzanne Stevenson. A probabilistic computational model of cross-situational word learning. Cognit. Sci., 2010. find
  14. Chuang Gan, Yandong Li, Haoxiang Li, Chen Sun, and Boqing Gong. VQS: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation. In ICCV, 2017. find
  15. Siddha Ganju, Olga Russakovsky, and Abhinav Gupta. What’s in a question: Using visual questions as a form of supervision. In CVPR, 2017. find
  16. Jon Gauthier, Roger Levy, and Joshua B Tenenbaum. Word learning and the acquisition of syntactic– semantic overhypotheses. In CogSci, 2018. find
  17. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In ICML, 2017.
  18. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2015.
  19. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017. find
  20. Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: learning abstract hierarchical compositional visual concepts. In ICLR, 2018. find
  21. Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In ECCV, 2018. find
  22. Drew A Hudson and Christopher D Manning. Compositional attention networks for machine reasoning. In ICLR, 2018.
  23. Allan Jabri, Armand Joulin, and Laurens van der Maaten. Revisiting visual question answering baselines. In ECCV, 2016. find
  24. Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In CVPR, 2015. find
  25. Justin Johnson, Andrej Karpathy, and Li Fei-Fei. DenseCap: Fully convolutional localization networks for dense captioning. In CVPR, 2016. find
  26. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017a.
  27. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In ICCV, 2017b.
  28. Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv:1411.2539, 2014.
  29. George Konidaris, Leslie Pack Kaelbling, and Tomas Lozano-Perez. From skills to symbols: Learning symbolic representations for abstract high-level planning. JAIR, 2018. find
  30. Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In NeurIPS, 2015. find
  31. Beth Levin. English verb classes and alternations, volume 1. University of Chicago Press, 1993. find
  32. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
  33. M Malinowski and M Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NeurIPS, 2014. find
  34. Junhua Mao, Jiajing Xu, Kevin Jing, and Alan L Yuille. Training and evaluating multimodal word embeddings with large-scale web annotated images. In NeurIPS, 2016. find
  35. David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar. Transparency by design: Closing the gap between performance and interpretability in visual reasoning. In CVPR, 2018. find
  36. Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In ICML, 2017. find
  37. Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
  38. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
  39. Adam Santoro, David Raposo, David GT Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In NeurIPS, 2017.
  40. Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D Manning. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In EMNLP Workshop, 2015. find
  41. Haoyue Shi, Jiayuan Mao, Tete Xiao, Yuning Jiang, and Jian Sun. Learning visually-grounded semantics from contrastive adversarial samples. In COLING, 2018. find
  42. N Siddharth, T. B. Paige, J.W. Meent, A. Desmaison, N. Goodman, P. Kohli, F. Wood, and P. Torr. Learning disentangled representations with semi-supervised deep generative models. In NeurIPS, 2017. find
  43. Joseph Suarez, Justin Johnson, and Fei-Fei Li. DDRprog: A clevr differentiable dynamic reasoning programmer. arXiv:1803.11361, 2018. — [arXiv:1803.11361](https://arxiv.org/abs/1803.11361) find
  44. Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NeurIPS, 2000. find
  45. Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language. In ICLR, 2016.
  46. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. MLJ, 8(3-4):229–256, 1992. find
  47. Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In CVPR, 2017. find
  48. Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In ECCV, 2016. find
  49. Jimei Yang, Scott E Reed, Ming-Hsuan Yang, and Honglak Lee. Weakly-supervised disentangling with recurrent transformations for 3d view synthesis. In NeurIPS, 2015. find
  50. Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In CVPR, 2016.
  51. Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenenbaum. Neural-Symbolic VQA: Disentangling reasoning from vision and language understanding. In NeurIPS, 2018.
  52. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In CVPR, 2015.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mao-2019-neuro-symbolic-concept-learner-interpreting/