Elamur Research
☰
home / papers / 2016 / Learning to Compose Neural Networks for Question Answering

Learning to Compose Neural Networks for Question Answering

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N16-1181.pdf

Summary

The authors propose a question answering system that works equally well with images and with structured knowledge bases. Instead of translating questions into a logical form, the model assembles a separate neural network for each question from small composable modules: find, relate, lookup, and, describe, exists, etc. The modules operate on attention distributions over pixels or entities, and all of them are differentiable. A separate structure predictor selects the best layout from candidates generated automatically from a syntactic parse of the question. The predictor and the module parameters are trained jointly using reinforcement learning. Training requires only (world, question, answer) triples; no annotation of network layouts is needed. The authors call this approach the dynamic neural module network (D-NMN). It achieves state-of-the-art results on the visual benchmark VQA and on the geographic corpus GeoQA. The work is significant because it combines the compositionality of semantic parsing with the flexibility of continuous representations. It also shows that a network whose structure adapts to the question generalizes better, even on very small datasets.

Key points

  • The model consists of two parts trained jointly: a layout selection model p(z|x) and an execution model that applies the assembled network to the world representation. Modules have two basic output types: Attention and Labels. The parameters of a given module are shared across all of its instances.
  • GeoQA is a small dataset of 263 examples about US geography. Evaluation used cross-validation, holding out one of the 10 environments at a time. D-NMN achieves an accuracy of 54.3. For comparison: NMN with a fixed structure scores 51.7, LSP-W 51, and LSP-F 48.
  • The authors created a new variant of the corpus, GeoQA+Q, which includes quantification: questions such as "is there…" require a yes/no answer. On it, D-NMN achieves 42.9 versus 35.7 for NMN, a relative improvement of about 20%.
  • On VQA, the model also outperforms previous results. No specific numbers are given here, because this part of the text is missing from the provided excerpt.
  • The code is published in the repository github.com/jacobandreas/nmn2.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

101 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 89 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1181 · arXiv 1601.01705

@inproceedings{andreas2016learning,
  title = {{Learning to Compose Neural Networks for Question Answering}},
  author = {Andreas, Jacob and Rohrbach, Marcus and Darrell, Trevor and Klein, Dan},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1181},
  eprint = {1601.01705},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n16-1181}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas, Andreas Vlachos, and Stephen Clark. 2013. Semantic parsing as machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria. find
  2. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In Proceedings of the Conference on Computer Vision and Pattern Recognition. find our card
  3. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In Proceedings of the International Conference on Computer Vision. find
  4. Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, volume 7, page 92. find
  5. Antoine Bordes, Sumit Chopra, and Jason Weston. 2014. Question answering with subgraph embeddings. Proceedings of the Conference on Empirical Methods in Natural Language Processing. Léon Bottou. 2014. From machine learning to machine reasoning. Machine learning, 94(2):133–149. find
  6. Marie-Catherine De Marneffe and Christopher D Manning. 2008. The Stanford typed dependencies representation. In Proceedings of the International Conference on Computational Linguistics, pages 1–8. find our card
  7. Edward Grefenstette. 2013. Towards a formal distributional semantics: Simulating logical calculi with tensors. Joint Conference on Lexical and Computational Semantics. find
  8. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1684–1692. Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard So… find our card
  9. Jayant Krishnamurthy and Thomas Kollar. 2013. Jointly learning to parse and perceive: connecting natural language to the physical world. Transactions of the Association for Computational Linguistics. find
  10. Jayant Krishnamurthy and Tom Mitchell. 2013. Vector space semantic parsing: A framework for compositional vector space models. In Proceedings of the ACL Workshop on Continuous Vector Space Models and their Compositionality. find
  11. Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2010. Inducing probabilistic CCG grammars from logical form with higher-order unification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1223–1233, Cambridge, Massachusetts. find
  12. Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  13. Percy Liang, Michael Jordan, and Dan Klein. 2011. Learning dependency-based compositional semantics. find
  14. In Proceedings of the Human Language Technology Conference of the Association for Computational Linguistics, pages 590–599, Portland, Oregon. Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. 2015. Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of t… find
  15. Cynthia Matuszek, Nicholas FitzGerald, Luke Zettlemoyer, Liefeng Bo, and Dieter Fox. 2012. A joint model of language and perception for grounded attribute learning. In International Conference on Machine Learning. find
  16. Hyeonwoo Noh, Paul Hongsuck Seo, and Bohyung Han. 2015. Image question answering using convolutional neural network with dynamic parameter prediction. arXiv preprint arXiv:1511.05756. — [arXiv:1511.05756](https://arxiv.org/abs/1511.05756) find
  17. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. In Advances in N… find
  18. K Simonyan and A Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. find our card
  19. Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013. Parsing with compositional vector grammars. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  20. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256. find
  21. Yuk Wah Wong and Raymond J. Mooney. 2007. Learning synchronous grammars for semantic parsing with lambda calculus. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, volume 45, page 960. find
  22. Huijuan Xu and Kate Saenko. 2015. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. arXiv preprint arXiv:1511.05234. — [arXiv:1511.05234](https://arxiv.org/abs/1511.05234) find
  23. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning. find our card
  24. Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. 2015. Stacked attention networks for image question answering. arXiv preprint arXiv:1511.02274. find
  25. Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao. 2015. Neural enquirer: Learning to query tables. arXiv preprint arXiv:1512.00965. — [arXiv:1512.00965](https://arxiv.org/abs/1512.00965) find
  26. Matthew D Zeiler. 2012. ADADELTA: An adaptive learning rate method. arXiv preprint arXiv:1212.5701. find our card
  27. Bolei Zhou, Yuandong Tian, Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. 2015. Simple baseline for visual question answering. arXiv preprint arXiv:1512.02167. — [arXiv:1512.02167](https://arxiv.org/abs/1512.02167) find