Elamur Research⌕
☰
home / papers / 2018 / Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks

Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks

Paper · 2018 · João Loula, Marco Baroni and Brenden Lake

Summary

The paper asks whether recurrent seq2seq models can generalize systematically by recombining words they already know well, as opposed to learning a brand-new word. The authors build on the SCAN benchmark (Lake and Baroni, 2018), in which simple navigation commands are translated into action sequences. They create new train/test splits in which every test word is frequent in training, but certain combinations, such as "jump around right", are withheld. Success therefore depends on exploiting the left/right symmetry and treating modifiers like "around" and "opposite" as second-order operators.

They reuse the strongest architecture from the original SCAN study, an LSTM seq2seq model, across three experiments. The model handles new combinations when the exact complex template has appeared with other fillers. It fails when it must derive a template's meaning from its parts, and it needs hundreds of in-context examples to reach peak accuracy.

The work shows that strong seq2seq generalization can come from analogy within familiar patterns rather than from learning compositional rules. This motivates research into inductive biases that would support systematicity.

Key points

  • Dataset and setup: SCAN has 12 base templates, combined through "and"/"after" and "twice"/"thrice" into 20,910 commands. The new splits are released in the SCAN GitHub repository.
  • Model and evaluation:
    • The model is a 2-layer, 200-unit LSTM with 50% dropout. It scored 99.8% on SCAN's random split, and an attention variant did worse in every condition.
    • Accuracy counts only fully correct output sequences and is averaged over 5 runs.
    • Every condition uses 100k training presentations.
    • Commands whose target involves "turn" were excluded because "turn" has distinct semantics.
  • Experiment 1 (four held-out templates):
    • Withholding only "jump around right" posed no difficulty.
    • Accuracy dropped sharply for more systematic gaps, including the shorter "Primitive right" pattern.
  • Experiment 2 (fillers of "Primitive around right"): Seeing the template with only one other primitive, roughly 1,100 extra examples, was enough for near-perfect transfer to "jump".
  • Experiment 3 (1 to 1,024 "look around right" examples): Accuracy climbed gradually, reaching about 70% at 64 examples and peaking at 512. This pattern suggests the model accumulates evidence case by case rather than extracting a rule.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States158 works · 211,459 citations🎓New York UniversityNew York · United States136 works · 86,390 citations🎓École PolytechniquePalaiseau · France5 works · 677 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as João Loula, Marco Baroni and Brenden Lake. 2018. Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-5413 · arXiv 1807.07545

@inproceedings{loula2018rearranging,
  title = {{Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks}},
  author = {João Loula, Marco Baroni and Brenden Lake},
  booktitle = {Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP},
  year = {2018},
  doi = {10.18653/v1/w18-5413},
  eprint = {1807.07545},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w18-5413}
}

References

16 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrew Kyle Lampinen and James L McClelland. 2017. One-shot and few-shot learning of word embeddings. CoRR. find
  2. Matthew Botvinick and David Plaut. 2009. Empirical and computational support for context-dependent representations of serial order: Reply to Bowers, Damian, and Davis (2009). Psychological Review, 116(4):998–1002. find
  3. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature, 521:436–444. find
  4. Gary F Marcus. 1998. Rethinking Eliminative Connectionism. Cognitive Psychology, 282(37):243–282. find
  5. Jeffrey Bowers, Markus Damian, and Colin David. 2009. A fundamental limitation of the conjunctive codes learned in PDP models of cognition: Comment on Botvinick and Plaut (2006). Psychological Review, 116(4):986–997. find
  6. Gary F Marcus. 2003. The Algebraic Mind: Integrating Connectionism and Cognitive Science. MIT Press, Cambridge, MA. find
  7. Richard Montague. 1970. Universal grammar. Theoria, 36(3):373–398. find
  8. Philémon Brakel and Stefan Frank. 2009. Strong systematicity in sentence processing by simple recurrent networks. In Proceedings of CogSci, pages 1599–1604, Amsterdam, the Netherlands. find
  9. Steven Phillips. 1998. Are feedforward and recurrent networks systematic? analysis and implications for a connectionist cognitive architecture. Connection Science, 10:137–160. find
  10. Frankilin Chang. 2002. Symbolically speaking: a connectionist model of sentence production. Cognitive Science, 26:609–651. find
  11. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  12. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259.
  13. Frank van der Velde, Gwendid van der Voort van der Kleij, and Marc de Kamps. 2004. Lack of combinatorial productivity in language processing with simple recurrent networks. Connection Science, 16(1):21–46. find
  14. Noam Chomsky. 1957. Syntactic structures. Walter de Gruyter. find
  15. Morten Christiansen and Nick Chater. 1994. Generalization and connectionist language learning. Mind & Language, 9(3):273–287. find
  16. Jerry A Fodor and Zenon W Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. Cognition, 28:3–71. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/loula-2018-rearranging-familiar-testing-compositional-generalization/