Elamur Research
☰
home / papers / 2020 / Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks

Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.tacl-1.9.pdf

Summary

When training data fit both a structure-based rule and an order-based rule, what properties of a neural learner decide which one it adopts? McCoy, Frank and Linzen test this with sequence-to-sequence networks on two synthetic English tasks: question formation and tense reinflection. A context-free grammar with a 68-word vocabulary generates the training sentences. These sentences support both a hierarchical rule (MOVE-MAIN) and a linear rule (MOVE-FIRST). Held-out examples with a relative clause on the subject then show which rule a model actually learned.

The authors varied three things: the recurrent unit (SRN, GRU, LSTM), the attention type, and sequential versus tree-structured processing. Every factor changed how models generalized, including factors with no obvious link to syntax. However, only tree-based models that received correct parses preferred hierarchical rules on both tasks. Sequential models failed even when given bracketed input or extra training tasks. The results suggest that human-like syntactic generalization from text alone may require explicit structure in the architecture. This bears on NLP model design and on the poverty-of-the-stimulus debate.

Key points

  • More than 3,500 models were trained, and the paper reports median results. Code is available at github.com/tommccoy1/rnn-hierarchical-biases.
  • Generalization was scored by first-word accuracy, meaning which auxiliary appears at the start of the question. Full-sentence outputs often contained unrelated errors. In over 98% of cases, most models began their questions with either the main-clause auxiliary or the sentence's earliest auxiliary.
  • LSTMs and GRUs showed qualitatively different biases. The authors link this partly to GRU hidden-state values being bounded within a fixed interval.
  • The architecture was not the only influence: interactions between the recurrent unit and the attention type also strongly shaped behaviour. The authors found no hyperparameter setting that made a non-tree model generalize hierarchically on both tasks.
  • Training setup: embeddings and hidden states of size 256, learning rate 0.001, batch size 5. Validation ran every 1,000 batches, with a minimum of 30,000 batches before early stopping.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

56 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 51 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00304

@article{mccoy2020does,
  title = {{Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks}},
  author = {McCoy, R. Thomas and Frank, Robert and Linzen, Tal},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.1162/tacl\_a\_00304},
  url = {https://doi.org/10.1162/tacl_a_00304}
}

References

40 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Noam Chomsky. 1965. Aspects of the Theory of Syntax, MIT Press, Cambridge, MA. We modified the structure of the gates rather than adding a squashing nonlinearity to avoid vanishing gradients. find
  2. Noam Chomsky. 1980. Rules and representations. Behavioral and Brain Sciences, 3(1):1–15. find
  3. Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-based models for speech recognition. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 1, pages 577–585. MIT Press. of RNNs with multi-task learning. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 3–14, Vancouver, Canada. Association for Computational Linguistics. find our card
  4. Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NeurIPS Deep Learning and Representation Learning Workshop. find
  5. Alexander Clark and Rémi Eyraud. 2007. Polynomial identification in the limit of substitutable context-free languages. Journal of Machine Learning Research, 8(Aug):1725–1745. find
  6. Hartmut Fitz and Franklin Chang. 2017. Meaningful questions: The acquisition of auxiliary inversion in a connectionist model of sentence production. Cognition, 166:225–250. find
  7. Jerry A. Fodor and Zenon W. Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1–2):3–71. find
  8. Robert Frank and Donald Mathis. 2007. Transformational networks. In Proceedings of the Workshop on Psychocomputational Models of Human Language Acquisition. Cognitive Science Society. find
  9. Alexander Clark and Shalom Lappin. 2010. Linguistic Nativism and the Poverty of the Stimulus, John Wiley & Sons. find
  10. Lila R. Gleitman and Henry Gleitman. 1992. A picture is worth a thousand words, but that’s the problem: The role of syntax in vocabulary acquisition. Current Directions in Psychological Science, 1(1):31–35. find
  11. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, pages 160–167. ACM. find
  12. Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. arXiv:1410.5401v2 preprint arXiv:1410.5401. ↗
  13. Stephen Crain and Mineharu Nakayama. 1987. Structure dependence in grammar formation. Language, pages 522–543. find
  14. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. ↗
  15. Emmanuel Dupoux. 2018. Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner. Cognition, 173:43–59. find
  16. Xuân-Nga Cao Kam, Iglika Stoyneshka, Lidiya Tornyova, Janet D. Fodor, and William G. Sakas. 2008. Bigrams and the richness of the stimulus. Cognitive Science, 32(4):771–787. find
  17. Chris Dyer, Gábor Melis, and Phil Blunsom. 2019. A critical analysis of biased parsers in unsupervised parsing. arXiv:1909.09428v1 preprint arXiv:1909.09428. ↗
  18. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. find
  19. Christo Kirov and Robert Frank. 2012. Processing of nested and cross-serial dependencies: An automaton perspective on SRN behaviour. Connection Science, 24(1):1–24. find
  20. Jeffrey L. Elman, Elizabeth A. Bates, Mark H. Johnson, Annette Karmiloff-Smith, Domenico Parisi, and Kim Plunkett. 1998. Rethinking innateness: A connectionist perspective on development. MIT press. find
  21. Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International Conference on Machine Learning, pages 2879–2888. find
  22. Émile Enguehard, Yoav Goldberg, and Tal Linzen. 2017. Exploring the syntactic abilities Barbara Landau, Linda B. Smith, and Susan S. Jones. 1988. The importance of shape in early lexical learning. Cognitive Development, 3(3): 299–321. find
  23. Steven Pinker. 1996. Language learnability and language development, with new commentary by the author, volume 7, Harvard University Press. find
  24. Stephen Laurence and Eric Margolis. 2001. The poverty of the stimulus argument. The British Journal for the Philosophy of Science, 52(2): 217–276. find
  25. Geoffrey K. Pullum and Barbara C. Scholz. 2002. Empirical assessment of stimulus poverty arguments. The Linguistic Review, 18(1–2): 9–50. find
  26. Julie Anne Legate and Charles D. Yang. 2002. Empirical re-assessment of stimulus poverty arguments. The Linguistic Review, 18(1–2):151–162. find
  27. John D. Lewis and Jeffrey L. Elman. 2001. Learnability and the statistical structure of language: Poverty of stimulus arguments revisited. In Proceedings of the 26th Annual Conference on Language Development. find
  28. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. find our card
  29. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421. Lisbon, Portugal. Association for Computational Linguistics. find
  30. Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019. Studying the inductive biases of RNNs with synthetic variations of natural languages. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3532–3542, Minneapolis, Minnesota. Association for Computational Linguistics. find
  31. Florencia Reali and Morten H. Christiansen. 2005. Uncovering the richness of the stimulus: Structure dependence and indirect statistical evidence. Cognitive Science, 29(6): 1007–1028. find
  32. Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. Proceedings of the 2019 International Conference on Learning Representations. find
  33. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. LSTM neural networks for language modeling. In Thirteenth Annual Conference of the International Speech Communication Association. find
  34. R. Thomas McCoy, Robert Frank, and Tal Linzen. 2018. Revisiting the poverty of the stimulus: Hierarchical generalization without a hierarchical bias in recurrent neural networks. In Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 2093–2098. Madison, WI. find
  35. R. Thomas McCoy, Tal Linzen, Ewan Dunbar, and Paul Smolensky. 2019. RNNs implicitly implement tensor-product representations. In Proceedings of the 2019 International Conference on Learning Representations. find
  36. James L. Morgan and Katherine Demuth. 1996. Signal to syntax: Bootstrapping from speech to grammar in early acquisition. Psychology Press. find
  37. Amy Perfors, Joshua B. Tenenbaum, and Terry Regier. 2011. The learnability of abstract syntactic principles. Cognition, 118(3):306–338. find
  38. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. find
  39. Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019. Can you tell me how to get past Sesame Street? Sentence-level pretraining beyond language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4465–4476. Florence, Italy. Association for Computational Linguistics. finite precision RNNs for language recognition. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 740–745. Association for Computational Linguistics. find
  40. Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of find