Elamur Research
☰
home / papers / 2015 / Neural Module Networks

Neural Module Networks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1511.02799

Summary

Visual question answering (VQA) means answering a natural-language question about an image. Many questions share parts: "where is the dog?" and "what color is the dog?" both require finding the dog. Earlier neural systems ignored this structure and encoded each question as a single vector. Andreas et al. instead propose neural module networks (NMNs). A semantic parser analyses each question, and the resulting structure is used to assemble a custom network for that question. The network is built from a small set of reusable, jointly trained modules, such as attending to an object, combining or shifting attention maps, and classifying or measuring. These modules pass image features, attention maps or labels between one another. An LSTM reading the whole question is added to capture dataset priors and common-sense knowledge. On the VQA benchmark the full model slightly surpasses the best published results. On a new synthetic dataset of complex questions about abstract shapes, it beats competing methods by up to 25 points of absolute accuracy. The work matters because it links symbolic semantic parsing with neural learning, pointing toward "programs" assembled from neural components.

Key points

  • Modules (e.g. attend, re-attend, combine, classify, measure) are typed by their inputs and outputs. They are composed dynamically per question, so a question can produce a layout never seen during training. Visual input is conv5 features of a 16-layer VGGNet, which is not fine-tuned.
  • The VQA dataset has more than 200,000 COCO images, each with three human-written questions and ten answers per question.
  • On the VQA test-dev split, NMN+LSTM scores 54.8% overall, compared with 53.7% for VIS+LSTM. On the test split it scores 55.1%, compared with 54.1%. NMN without the LSTM reaches only 42.7%.
  • The model is strongest on number and "other" (object or attribute) questions, with 37.2% and 39.3%. It is weaker on yes/no questions (77.7% vs. 78.9% for VIS+LSTM), which the authors attribute to overfitting in the measure module.
  • The authors hand-checked 50 parses. About 80–90% of simple property questions were analysed correctly, while complex questions often picked up irrelevant predicates. A better parser, or learning the parser jointly with the modules, is identified as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

167 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 155 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.02799 · arXiv 1511.02799

@misc{andreas2015neural,
  title = {{Neural Module Networks}},
  author = {Andreas, Jacob and Rohrbach, Marcus and Darrell, Trevor and Klein, Dan},
  year = {2015},
  doi = {10.48550/arxiv.1511.02799},
  eprint = {1511.02799},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.02799}
}

References

33 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Andreas and D. Klein. Grounding language with points and paths in continuous spaces. CoNLL, 2014. 3 find
  2. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. In ICCV, 2015. 1, 2, 3, 7 find
  3. K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014. 3 find our card
  4. M.-C. De Marneffe and C. D. Manning. The Stanford typed dependencies representation. In Proceedings of the International Conference on Computational Linguistics, pages 1–8. Association for Computational Linguistics, 2008. 4 find our card
  5. J. L. Elman. Finding structure in time. Cognitive science, 14(2):179–211, 1990. 3 find
  6. N. FitzGerald, Y. Artzi, and L. Zettlemoyer. Learning distributions over logical forms for referring expression generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2013. 2 find
  7. H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu. Are you talking to a machine? dataset and methods for multilingual image question answering. In NIPS, 2015. 1, 3 find
  8. D. Geman, S. Geman, N. Hallonquist, and L. Younes. Visual turing test for computer vision systems. Proceedings of the National Academy of Sciences, 2015. 2 find
  9. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 580–587. IEEE, 2014. 2 find our card
  10. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 3 find
  11. M. Iyyer, J. Boyd-Graber, L. Claudino, R. Socher, and H. Daumé III. A neural network for factoid question answering over paragraphs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2014. 2 find
  12. A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015. 3 find our card
  13. A. Karpathy, A. Joulin, and L. Fei-Fei. Deep fragment embeddings for bidirectional image sentence mapping. In NIPS, 2014. 3 find
  14. D. Klein and C. D. Manning. Accurate unlexicalized parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 423–430. Association for Computational Linguistics, 2003. 4 find our card
  15. C. Kong, D. Lin, M. Bansal, R. Urtasun, and S. Fidler. What are you talking about? text-to-image coreference. In CVPR, pages 3558–3565. IEEE, 2014. 3 find
  16. J. Krishnamurthy and T. Kollar. Jointly learning to parse and perceive: connecting natural language to the physical world. Transactions of the Association for Computational Linguistics, 2013. 1, 3, 8 find
  17. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. 7 find
  18. P. Liang, M. I. Jordan, and D. Klein. Learning dependency-based compositional semantics. Computational Linguistics, 39(2):389–446, 2013. 1, 3, 5 find
  19. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014, pages 740–755. Springer, 2014. 2 find our card
  20. L. Ma and Z. L. andiyyer Hang Li. Learning to answer questions from image using convolutional neural network. arXiv:1506.00333, 2015. 1, 3 — [arXiv:1506.00333](https://arxiv.org/abs/1506.00333) find
  21. M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, 2014. 1, 2, 3 find
  22. M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions about images. In ICCV, 12/2015 2015. 1, 2 find
  23. C. Matuszek, N. Fitzgerald, L. Zettlemoyer, L. Bo, and D. Fox. A joint model of language and perception for grounded attribute learning. In ICML, 2012. 3 find
  24. B. Plummer, L. Wang, C. Cervantes, J. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV, 2015. 3 find
  25. M. Ren, R. Kiros, and R. Zemel. Image question answering: A visual semantic embedding model and a new dataset. In NIPS, 2015. 1, 3 find
  26. M. Ren, R. Kiros, and R. S. Zemel. Image question answering: A visual semantic embedding model and a new dataset. CoRR, abs/1505.02074, 2015. 7 find
  27. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014. 7 find our card
  28. R. Socher, J. Bauer, C. D. Manning, and A. Y. Ng. Parsing with compositional vector grammars. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2013. 3 find
  29. J. Weston, A. Bordes, S. Chopra, and T. Mikolov. Towards ai-complete question answering: a set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015. 6 find our card
  30. J. Weston, S. Chopra, and A. Bordes. Memory networks. arXiv preprint arXiv:1410.3916, 2014. 3 find
  31. K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044, 2015. 3 find our card
  32. L. Yu, E. Park, A. C. Berg, and T. L. Berg. Visual madlibs: Fill in the blank image generation and question answering. arXiv:1506.00278, 2015. 1, 2 — [arXiv:1506.00278](https://arxiv.org/abs/1506.00278) find
  33. M. D. Zeiler. ADADELTA: An adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012. 6 find our card