Elamur Research⌕
☰
home / papers / 2017 / Inferring and Executing Programs for Visual Reasoning

Inferring and Executing Programs for Visual Reasoning

Paper · 2017 · Justin Johnson, Bharath Hariharan, Laurens van der Maaten et al.

Summary

Visual question answering systems usually learn a direct mapping from image and question to answer. Such black-box models tend to exploit dataset biases instead of actually reasoning. The authors argue that compositional reasoning should be built into the model's structure, and they split the task into two learned parts. A program generator, an LSTM sequence-to-sequence model, turns the question into a program: a sequence of functions that encodes a syntax tree in prefix order. An execution engine then assembles a neural module network that mirrors this program and runs it on CNN image features to predict an answer. Unlike earlier module networks, neither part relies on hand-built syntactic parsers or hand-designed modules. Only the function vocabulary and a generic module design are fixed. Training uses backpropagation, with REINFORCE for joint end-to-end tuning. On CLEVR, the model beats strong non-compositional baselines by about 20 points using little program supervision. It also adapts to free-form questions written by humans. This shows that explicit, interpretable reasoning structures generalize better than direct mappings.

Key points

  • Evaluated on the CLEVR benchmark. Only about 9,000 ground-truth programs (roughly 2% of those available) were needed to outperform state-of-the-art VQA models by around 20 percentage points.
  • Programs are built from a fixed set of functions of arity 1 or 2, plus a special Scene constant for image features. Malformed predicted sequences are repaired by padding with Scene or by dropping unused functions.
  • The authors collected a new dataset of human-written free-form questions about CLEVR images. After finetuning without extra program supervision, the model gained about 9 accuracy points over the best competing models.
  • Reimplemented baselines include LSTM, CNN+LSTM, CNN+LSTM+SA and CNN+LSTM+SA+MLP. Adding an MLP head to the stacked-attention model raised CLEVR accuracy from 69.8 to 73.2, and shape-comparison accuracy from 50.9 to 69.7.
  • The dependency-parser layouts from Andreas et al.'s dynamic module networks often failed on long CLEVR questions. In those cases they fell back to the default fragment "(what thing)".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations🏢MetaMenlo Park · United States214 works · 241,492 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick and Ross Girshick. 2017. Inferring and Executing Programs for Visual Reasoning. arXiv:1705.03633.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1705.03633 · arXiv 1705.03633

@misc{johnson2017inferring,
  title = {{Inferring and Executing Programs for Visual Reasoning}},
  author = {Justin Johnson, Bharath Hariharan, Laurens {van der Maaten}, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick and Ross Girshick},
  year = {2017},
  doi = {10.48550/arxiv.1705.03633},
  eprint = {1705.03633},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1705.03633}
}

References

51 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Learning to compose neural networks for question answering. In NAACL, 2016. 1, 2, 3, 5, 8, 9, 10, 11
  2. J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks. In CVPR, 2016. 1, 2, 3, 4, 5, 8, 9
  3. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Zitnick, and D. Parikh. VQA: Visual question answering. In ICCV, 2015. 1, 2, 4, 7, 11 find
  4. M. Balog, A. Gaunt, M. Brockschmidt, S. Nowozin, and D. Tarlow. Deepcoder: Learning to write programs. In ICLR, 2017. 2, 8 find
  5. J. Cai, R. Shin, and D. Song. Making neural programming architectures generalize via recursion. In ICLR, 2017. 8 find
  6. A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. Moura, D. Parikh, and D. Batra. Visual dialog. In CVPR, 2017. 2 find
  7. J. Devlin, S. Gupta, R. Girshick, M. Mitchell, and C. L. Zitnick. Exploring nearest neighbor approaches for image captioning. arXiv preprint arXiv:1505.04467, 2015. 1 find
  8. A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In arXiv:1606.01847, 2016. 2
  9. Y. Gao, O. Beijbom, N. Zhang, and T. Darrell. Compact bilinear pooling. In CVPR, 2016. 2 find
  10. R. Girshick. Fast R-CNN. In ICCV, 2015. 1 find
  11. Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. 2
  12. A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. 2 find
  13. A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Barwinska, S. Colmenarejo, E. Grefenstette, T. Ramalho, J. Agapiou, A. Badia, K. Hermann, Y. Zwols, G. Ostrovski, A. Cain, H. King, C. Summerfield, P. Blunsom, K. . Kavukcuoglu, and D. Hassabis. Hybrid computing using a neural net… find
  14. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016. 3, 9
  15. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997. 4 find
  16. R. Hu, M. Rohrbach, J. Andreas, T. Darrell, and K. Saenko. Modeling relationships in referential expressions with compositional modular networks. In CVPR, 2017. 2 find
  17. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. 9 find
  18. A. Jabri, A. Joulin, and L. van der Maaten. Revisiting visual question answering baselines. In ECCV, 2016. 1, 2 find
  19. J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 1, 2, 4, 5, 6, 7, 9, 10
  20. A. Joulin and T. Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In NIPS, 2015. 2 find
  21. K. Kafle and C. Kanan. Visual question answering: Datasets, algorithms, and future challenges. In arXiv preprint arXiv:1610.01465, 2016. 2 find
  22. Ł. Kaiser and I. Sutskever. Neural GPUs learn algorithms. In ICLR, 2016. 2 find
  23. D. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 9
  24. D. Klein and C. D. Manning. Accurate unlexicalized parsing. In ACL, pages 423–430, 2003. 2
  25. R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017. 2
  26. A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS. 2012. 1
  27. K. Kurach, M. Andrychowicz, and I. Sutskever. Neural random-access machines. In ICLR, 2016. 2 find
  28. B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 2016. 1 find
  29. C. Liang, J. Berant, Q. Le, K. D. Forbus, and N. Lao. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. arXiv preprint arXiv:1611.00020, 2016. 2, 8 find
  30. P. Liang, M. I. Jordan, and D. Klein. Learning dependency-based compositional semantics. In ACL, 2011. 2 find
  31. J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical question-image co-attention for visual question answering. In NIPS, 2016. 2 find
  32. M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, 2014. 2 find
  33. M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions about images. In ICCV, 2015. 2, 4 find
  34. A. Mallya and S. Lazebnik. Learning models for actions and person-object interactions with transfer to question answering. In ECCV, 2016. 2 find
  35. A. Neelakantan, Q. V. Le, and I. Sutskever. Neural programmer: Inducing latent programs with gradient descent. In ICLR, 2016. 2 find
  36. S. Reed and N. De Freitas. Neural programmer-interpreters. In ICLR, 2016. 2, 8 find
  37. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. ImageNet large scale visual recognition challenge. IJCV, 2015. 3, 9
  38. S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus. End-to-end memory networks. In NIPS, 2015. 2 find
  39. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In NIPS, 2014. 3, 9
  40. M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler. Movieqa: Understanding stories in movies through question-answering. In CVPR, 2016. 2 find
  41. J. Weston, S. Chopra, and A. Bordes. Memory networks. In ICLR, 2015. 2 find
  42. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(23), 1992. 4 find
  43. T. Winograd. Understanding Natural Language. Academic Press, 1972. 2 find
  44. Q. Wu, D. Teney, P. Wang, C. Shen, A. Dick, and A. van den Hengel. Visual question answering: A survey of methods and datasets. In arXiv preprint arXiv:1607.05910, 2016. 2 — [arXiv:1607.05910](https://arxiv.org/abs/1607.05910) find
  45. C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering. ICML, 2016. 2 find
  46. Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering. In CVPR, 2016. 4, 5, 10
  47. W. Zaremba, T. Mikolov, A. Joulin, and R. Fergus. Learning simple algorithms from examples. In ICML, 2016. 2 find
  48. W. Zaremba and I. Sutskever. Learning to execute. arXiv preprint arXiv:1410.4615, 2014. 2 find
  49. W. Zaremba and I. Sutskever. Reinforcement learning neural turing machines. arXiv preprint arXiv:1505.00521, 2015. 2
  50. P. Zhang, Y. Goyal, D. Summers-Stay, D. Batra, and D. Parikh. Yin and yang: Balancing and answering binary visual questions. In CVPR, 2016. 2 find
  51. Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. Visual7W: Grounded question answering in images. In CVPR, 2016. 1, 2

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/johnson-2017-inferring-executing-programs-visual-reasoning/