Elamur Research⌕
☰
home / papers / 2018 / Systematic Generalization: What Is Required and Can It Be Learned?

Systematic Generalization: What Is Required and Can It Be Learned?

Paper · 2018 · Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch et al.

Summary

The authors test whether neural models of grounded language understanding can generalize systematically, that is, correctly understand new combinations of words they have already seen. To do this, they created a synthetic VQA dataset, SQOOP. Each 64×64 image contains five randomly placed letters and digits, and the model answers "yes" or "no" to a query of the form X R Y about the spatial relation between two objects. Models are trained on only some of the object pairs and tested on the rest. The comparison includes generic models (CNN+LSTM, FiLM, RelNet, MAC) and modular Neural Module Networks (NMN) whose modules are connected in different ways: as a chain or as a tree. NMNs generalize noticeably better, but only when the layout matches the structure of the task, i.e., when it is tree-structured. The chain variants fail on the hardest version of the dataset. Methods that learn the layout or the module parameterization end-to-end (Johnson et al., Hu et al.) often converge to poor solutions. The paper concludes that modularity alone is not enough: explicit regularizers or priors are also needed. This matters for anyone building robust language understanding systems.

Key points

  • SQOOP has 36 objects (A–Z, 0–9) and 4 relations (LEFT OF, RIGHT OF, ABOVE, BELOW), giving 5040 unique questions in total. Each training set contains 1M examples.
  • Difficulty is set by the parameter #rhs/lhs. This is the number of different right-hand objects for each left-hand object in the training data: 1, 2, 4, 8, 18, with 35 used as a control. The lower the value, the more spurious correlations between words and the harder it is to generalize.
  • MAC at #rhs/lhs=1 gives unstable results. The standard configuration (12 cells, dimension 128) yields a test error of 13.67 ± 9.97 %, while with dimension 256 the error drops to 4.62 %. The runs that do generalize show sharp attention to each of the two object words.
  • A Residual NMN that learned a mixed layout (half tree, half chain) has a low error of 1.64 ± 1.79 %. However, its tree branch reaches about 99 % accuracy, while its chain branch reaches only about 60 %.
  • End-to-end NMNs often fail to arrive at a tree-structured layout or learn a "blurred" module parameterization. As a result, they generalize poorly on the hardest versions of SQOOP.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de MontréalMontreal · Canada87 works · 91,597 citations🏢Google DeepMindLondon · United Kingdom72 works · 84,326 citations🔬Canadian Institute for Advanced ResearchToronto · Canada35 works · 19,316 citations🎓University of OregonEugene · United States26 works · 3,555 citations🎓Indian Institute of Technology IndoreIndore · India2 works · 1,192 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries and Aaron Courville. 2018. Systematic Generalization: What Is Required and Can It Be Learned?. arXiv:1811.12889.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1811.12889 · arXiv 1811.12889

@misc{bahdanau2018systematic,
  title = {{Systematic Generalization: What Is Required and Can It Be Learned?}},
  author = {Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm {de Vries} and Aaron Courville},
  year = {2018},
  doi = {10.48550/arxiv.1811.12889},
  eprint = {1811.12889},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.12889}
}

References

35 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the Behavior of Visual Question Answering Models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, January 2016.
  2. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural Module Networks. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. URL http://arxiv.org/abs/1511.02799. ↗
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the 2015 International Conference on Learning Representations, 2015.
  4. Joost Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. Jump to better conclusions: SCAN both left and right. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 47–55, Brussels, Belgium, November 2018. Association for Co…
  5. Eli Bingham, Piero Molino, Paul Szerlip, Obermeyer Fritz, and Goodman Noah. Characterizing how Visual Question Answering scales with the world. In NIPS 2017 Visually-Grounded Interaction and Language Workshop, 2017. find
  6. Francisco Calvo and Eliana Colunga. The statistical brain: Reply to Marcus The algebraic mind. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 25, 2003. find
  7. Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1):3–71, 1988. find
  8. Alexander L. Gaunt, Marc Brockschmidt, Nate Kushman, and Daniel Tarlow. Differentiable Programs with Neural Libraries. In Proceedings of the 34th International Conference on Machine Learning, November 2016. URL http://arxiv.org/abs/1611.02109. arXiv: 1611.02109. — [arXiv:1611.02109](https://arxiv.org/abs/1611.02109) ↗
  9. Yichen Gong, Heng Luo, and Jian Zhang. Natural Language Inference over Interaction Space. In Proceedings of the 2018 International Conference on Learning Representations, 2017. URL http://arxiv.org/abs/1709.04348. arXiv: 1709.04348. ↗
  10. Nitish Gupta and Mike Lewis. Neural Compositional Denotational Semantics for Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018. URL http://aclweb.org/anthology/D18-1239. ↗
  11. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation Artifacts in Natural Language Inference Data. In Proceedings of NAACL-HLT 2018, March 2018. URL http://arxiv.org/abs/1803.02324. arXiv: 1803.02324. ↗
  12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  13. Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to Reason: End-to-End Module Networks for Visual Question Answering. In Proceedings of 2017 IEEE International Conference on Computer Vision, April 2017. URL http://arxiv.org/abs/1704.05526. arXiv: 1704.05526. ↗
  14. Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable Neural Computation via Stack Neural Module Networks. In Proceedings of 2018 European Conference on Computer Vision, July 2018. URL http://arxiv.org/abs/1807.08556. arXiv: 1807.08556. — [arXiv:1807.08556](https://arxiv.org/abs/1807.08556) ↗
  15. Drew A. Hudson and Christopher D. Manning. Compositional Attention Networks for Machine Reasoning. In Proceedings of the 2018 International Conference on Learning Representations, February 2018. URL https://openreview.net/forum?id=S1Euwz-Rb. ↗
  16. Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 448–456, 2015. URL http://jmlr.org/proceedings/pap… ↗
  17. Robin Jia and Percy Liang. Adversarial Examples for Evaluating Reading Comprehension Systems. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2021–2031, 2017. doi: 10.18653/v1/D17-1215. URL https://aclanthology.coli. uni-saarland.de/papers/D17-1215/d17-121… ↗
  18. Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Pythia v0.1: The winning entry to the vqa challenge 2018. https://github.com/facebookresearch/pythia, 2018. ↗
  19. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), December 20…
  20. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Inferring and Executing Programs for Visual Reasoning. In Proceedings of 2017 IEEE International Conference on Computer Vision, 2017. URL http://arxiv.org/abs/1705.03633. ↗
  21. Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, Laszlo Lukacs, Marina Ganea, Peter Young, and Vivek Ramavajjala. Smart Reply: Automated Response Suggestion for Email. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge D… find
  22. Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the 2015 International Conference on Learning Representations, 2015. URL http://arxiv.org/abs/1412.6980. arXiv: 1412.6980. ↗
  23. Alexander Kuhnle and Ann Copestake. ShapeWorld - A new test methodology for multimodal language understanding. arXiv:1704.04517 [cs], April 2017. URL http://arxiv.org/abs/1704.04517. arXiv: 1704.04517. — [arXiv:1704.04517](https://arxiv.org/abs/1704.04517) ↗
  24. Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 36th International Conference on Machine Learning, 2018. URL http://arxiv.org/abs/1711.00350. arXiv: 1711.00350. ↗
  25. Joao Loula, Marco Baroni, and Brenden M. Lake. Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks. In Proceedings of the 2018 BlackboxNLP EMNLP Workshop, July 2018. URL https://arxiv.org/abs/1807.07545. ↗
  26. Mateusz Malinowski and Mario Fritz. A Multi-world Approach to Question Answering About Real-world Scenes Based on Uncertain Input. In Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS’14, pp. 1682–1690, Cambridge, MA, USA, 2014. MIT Press. URL http://dl.… ↗
  27. Gary F. Marcus. Rethinking Eliminative Connectionism. Cognitive Psychology, 37(3):243–282, December 1998. ISSN 0010-0285. doi: 10.1006/cogp.1998.0694. URL http://www.sciencedirect.com/science/article/pii/S0010028598906946. — [doi:10.1006/cogp.1998.0694](https://doi.org/10.1006/cogp.1998.0694) ↗
  28. Gary F. Marcus. The algebraic mind: Integrating connectionism and cognitive science. MIT press, 2003. find
  29. Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual Reasoning with a General Conditioning Layer. In In Proceedings of the 2017 AAAI Conference on Artificial Intelligence, 2017. URL http://arxiv.org/abs/1709.07871. ↗
  30. Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems 31, June 2017. URL http://arxiv.org/abs/1706.01427. arXiv: 1706.… ↗
  31. Paul Smolensky. The constituent structure of connectionist mental states: A reply to Fodor and Pylyshyn. Southern Journal of Philosophy, 26(Supplement):137–161, 1987. find
  32. Joseph Suarez, Justin Johnson, and Fei-Fei Li. DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer. arXiv:1803.11361 [cs], March 2018. URL http://arxiv.org/abs/1803.11361. arXiv: 1803.11361. — [arXiv:1803.11361](https://arxiv.org/abs/1803.11361) ↗
  33. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems 27, pp. 3104–3112, 2014.
  34. Wei Wang, Ming Yan, and Chen Wu. Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1705–1714, Melbourne, Australia, 2018. A… find
  35. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, and others. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint arXiv:1609.08144, 2016. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bahdanau-2018-systematic-generalization-required-learned/