Elamur Research⌕
☰
home / papers / 2018 / Jump to better conclusions: SCAN both left and right

Jump to better conclusions: SCAN both left and right

Paper · 2018 · Jasmijn Bastings, Marco Baroni, Jason Weston et al.

Summary

This BlackboxNLP 2018 paper re-examines SCAN, the command-to-action benchmark from Lake and Baroni (2018) that tests compositional generalization in recurrent sequence-to-sequence models. The authors argue that SCAN rewards overly simple models. Because its grammar is shallow, the outputs have almost no target-side dependencies. A decoder can therefore succeed by moving a pointer across the input and translating piece by piece, with no real composition.

As a complement, they introduce NACS, which reverses the direction and maps action sequences back to commands. This mapping is one-to-many and forces the model to keep track of what it has already generated.

They evaluate simple RNN, GRU and LSTM encoder-decoders, with and without Bahdanau attention. An ablation ("-Dep") cuts the decoder off from its previous outputs. They also rerun all of Lake and Baroni's experiments, including the English-French MT probe, using early stopping on held-out validation data.

Ablated models remain strong on SCAN but drop sharply on NACS. Even full attention models generalize poorly on NACS. The authors therefore present NACS as a more demanding benchmark for future work.

Key points

  • SCAN contains 20,910 unambiguous commands. It is tested in simple, length and primitive (turn left, jump) splits. For each split, 10% of training data was held out as a validation set for early stopping.
  • On the simple split, every model scores roughly 100% on SCAN. Without access to previous outputs, NACS accuracy drops to 61.1 (RNN +Attn -Dep) and 51.2 (GRU +Attn -Dep), compared with 99.8 for the full models.
  • On "turn left", the simpler -Dep attention models (92.0 and 90.8) beat their full counterparts on SCAN. On "jump", GRU +Attn reaches 12.5, compared with the 1.2 reported by L&B. Attention helped in every setting, and generalizing to longer sequences remained hard.
  • In the "daxy" MT probe, a GRU with mini-batching and early stopping translated 70.8% (En-Fr) and 54.2% (Fr-En) of the novel sentences correctly. L&B had reported 12.5%.
  • Code and data are available at github.com/facebookresearch/NACS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States183 works · 228,558 citations🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations🏢Meta (United Kingdom)London · United Kingdom46 works · 18,033 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho and Douwe Kiela. 2018. Jump to better conclusions: SCAN both left and right. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-5407 · arXiv 1809.04640

@inproceedings{bastings2018jump,
  title = {{Jump to better conclusions: SCAN both left and right}},
  author = {Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho and Douwe Kiela},
  booktitle = {Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP},
  year = {2018},
  doi = {10.18653/v1/w18-5407},
  eprint = {1809.04640},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w18-5407}
}

References

19 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the International Conference on Learning Representations (ICLR), San Diego, USA.
  2. Adam Liska, Germán Kruszewski, and Marco Baroni. 2018. Memorize or generalize? searching for a compositional RNN in a haystack. CoRR, abs/1802.06467. — [arXiv:1802.06467](https://arxiv.org/abs/1802.06467) find
  3. Philémon Brakel and Stefan Frank. 2009. Strong systematicity in sentence processing by simple recurrent networks. In 31th Annual Conference of the Cognitive Science Society (COGSCI-2009), pages 1599–1604. Cognitive Science Society. find
  4. Tomas Mikolov, Armand Joulin, and Marco Baroni. 2016. A roadmap towards machine intelligence. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 29– 61. Springer. find
  5. Steven Phillips. 1998. Are feedforward and recurrent networks systematic? analysis and implications for a connectionist cognitive architecture. Connection Science, 10(2):137–160. find
  6. Rich Caruana, Steve Lawrence, and C Lee Giles. 2001. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In Advances in neural information processing systems, pages 402– 408. find
  7. Paul Rodriguez and Janet Wiles. 1998. Recurrent neural networks can learn to implement symbol-sensitive counting. In Advances in Neural Information Processing Systems, pages 87–93. find
  8. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, et al. 2018. The best of both worlds: Combining recent advances in neural machine translation. arXiv preprint arXiv:1804.09849.
  9. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112.
  10. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder– Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langu…
  11. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010.
  12. Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision rnns for language recognition. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Melbourne, Australia. Association for Computat…
  13. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. find
  14. Jerry A Fodor and Zenon W Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3–71. find
  15. Francis CK Wong and William SY Wang. 2007. Generalisation towards combinatorial productivity in language acquisition by simple recurrent networks. In Integration of Knowledge Intensive Multi-Agent Systems, 2007. find
  16. KIMAS 2007. International Conference on, pages 139–144. IEEE. find
  17. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122.
  18. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735–1780. find
  19. Łukasz Kaiser and Samy Bengio. 2016. Can active memory replace attention? In Advances in Neural Information Processing Systems, pages 3781–3789. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bastings-2018-jump-better-conclusions-scan-both/