Elamur Research
☰
home / papers / 2016 / Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/Q16-1037.pdf

Summary

Recurrent networks have no built-in model of syntax, so it is unclear whether they can learn rules that depend on sentence structure rather than on word order. The authors test this with English subject-verb number agreement. To choose the right verb form, a model must find the true subject and ignore other nouns that come in between, especially "attractors", which are nouns with the opposite number. They collected about 1.35 million agreement instances from Wikipedia and trained LSTMs under three kinds of supervision: predicting the verb's number from the words before it, making sentence-level grammaticality judgments, and plain language modeling. With direct supervision, the LSTM got almost all cases right. Its accuracy dropped as attractors were added, but it still did far better than structure-blind baselines. The language-modeling objective alone generalized poorly on the difficult cases. So LSTMs can approximate hierarchical dependencies when given targeted signals, but predicting the next word is not enough. The paper also shows why models should be tested on samples chosen for grammatical difficulty, not only on random text.

Key points

  • Data and split: about 1.35M number-prediction examples from Wikipedia, split 9% train, 1% validation and 90% test. The very large test set ensures that rare constructions are covered.
  • Model: one-hot word inputs, 50-dimensional embeddings, an LSTM with 50 hidden units and a logistic-regression output. Overall error was under 1%. With four intervening attractors, error rose to about 17%, while the baselines did much worse than chance.
  • Where errors happened: mistakes were most frequent when overt cues such as function words were missing, for example in noun-noun compounds and reduced relative clauses.
  • Weaker supervision: training only on sentence-level grammaticality labels raised errors only slightly. A language-modeling objective performed below chance on hard cases. Even the large-scale language model of Jozefowicz et al. (2016) made over five times as many errors as the number-prediction model on these cases.
  • Recommendation: add explicit syntactic signals to language-model objectives (e.g., through multi-task training) for applications that depend on structure, such as machine translation or text generation. Published in TACL, vol. 4 (2016).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

502 papers of our library cite it (of the 140673 whose bibliographies we read); 42 of them have a page here, 460 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00115 · arXiv 1611.01368

@misc{linzen2016assessing,
  title = {{Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies}},
  author = {Linzen, Tal and Dupoux, Emmanuel and Goldberg, Yoav},
  year = {2016},
  doi = {10.1162/tacl\_a\_00115},
  eprint = {1611.01368},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00115}
}

References

45 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joseph Allen and Mark S. Seidenberg. 1999. The emergence of grammaticality in connectionist networks. In Brian MacWhinney, editor, Emergentist approaches to language: Proceedings of the 28th Carnegie symposium on cognition, pages 115–151. find
  2. Mahwah, NJ: Erlbaum. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference for Learning Representations. find our card
  3. Emily M. Bender, Dan Flickinger, Stephan Oepen, and Yi Zhang. 2011. Parser evaluation over local and non-local deep dependencies in a large corpus. In Proceedings of EMNLP, pages 397–408. find our card
  4. Kathryn Bock and Carol A. Miller. 1991. Broken agreement. Cognitive Psychology, 23(1):45–93. find
  5. Melissa Bowerman. 1988. The “no negative evidence” problem: How do children avoid constructing an overly general grammar? In John A. Hawkins, editor, Explaining language universals, pages 73–101. find
  6. Oxford: Basil Blackwell. Samuel R. Bowman, Christopher D. Manning, and Christopher Potts. 2015. Tree-structured composition in neural networks without tree-structured architectures. In Proceedings of the NIPS Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches. find
  7. Bo Cartling. 2008. On the implicit acquisition of a context-free grammar by a simple recurrent neural network. Neurocomputing, 71(7):1527–1537. find
  8. Rich Caruana. 1998. Multitask learning. In Sebastian Thrun and Lorien Pratt, editors, Learning to learn, pages 95–133. find
  9. Boston: Kluwer. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005. find our card
  10. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of EMNLP, pages 1724–1734. find
  11. Noam Chomsky. 1965. Aspects of the Theory of Syntax. Cambridge, MA: MIT press. find
  12. Alexander Clark, Gianluca Giorgolo, and Shalom Lappin. 2013. Statistical representation of grammaticality judgements: The limits of n-gram models. In Proceedings of the Fourth Annual Workshop on Cognitive Modeling and Computational Linguistics (CMCL), pages 28–36. find
  13. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and A. Noah Smith. 2016. Recurrent neural network grammars. In Proceedings of NAACL/HLT, pages 199–209. find
  14. Kathleen M. Eberhard, J. Cooper Cutting, and Kathryn Bock. 2005. Making syntax of sense: Number agreement in sentence production. Psychological Review, 112(3):531–559. find
  15. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. find
  16. Jeffrey L. Elman. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7(2-3):195–225. find
  17. Jeffrey L. Elman. 1993. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71–99. find
  18. Martin B. H. Everaert, Marinus A. C. Huybregts, Noam Chomsky, Robert C. Berwick, and Johan J. Bolhuis. 2015. Structures, not strings: Linguistics as part of the cognitive sciences. Trends in Cognitive Sciences, 19(12):729–743. find
  19. Robert Frank, Donald Mathis, and William Badecker. 2013. The acquisition of anaphora by simple recurrent networks. Language Acquisition, 20(3):181–227. Felix Gers and Jürgen Schmidhuber. 2001. LSTM recurrent networks learn simple context-free and context-sensitive languages. IEEE Transactions on Ne… find
  20. Anastasia Giannakidou. 2011. Negative and positive polarity items: Variation, licensing, and compositionality. In Claudia Maienborn, Klaus von Heusinger, and Paul Portner, editors, Semantics: An international handbook of natural language meaning. find
  21. Berlin: Mouton de Gruyter. Yoav Goldberg and Joakim Nivre. 2012. A dynamic oracle for arc-eager dependency parsing. In Proceedings of COLING 2012, pages 959–976. find
  22. Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. 2015. Learning to transduce with unbounded memory. In Advances in Neural Information Processing Systems, pages 1828–1836. find
  23. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. find
  24. Rodney Huddleston and Geoffrey K. Pullum. 2002. The Cambridge Grammar of the English Language. find our card
  25. Cambridge University Press, Cambridge. Armand Joulin and Tomas Mikolov. 2015. Inferring algorithmic patterns with stack-augmented recurrent nets. In Advances in Neural Information Processing Systems, pages 190–198. find
  26. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. Ákos Kádár, Grzegorz Chrupała, and Afra Alishahi. 2016. Representation of linguistic form and function in recurrent neural networks. arXi… find
  27. Andrej Karpathy, Justin Johnson, and Fei-Fei Li. 2016. Visualizing and understanding recurrent networks. In ICLR Workshop. find our card
  28. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference for Learning Representations. find
  29. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional lstm feature representations. Transactions of the Association of Computational Linguistics, 4:313–327. find our card
  30. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2015. Unsupervised prediction of acceptability judgements. In Proceedings of ACL/IJCNLP, pages 1618–1628. find
  31. Steve Lawrence, Lee C. Giles, and Santliway Fong. 1996. Can recurrent neural networks learn natural language grammars? In IEEE International Conference on Neural Networks, volume 4, pages 1853–1858. find
  32. Willem J. M. Levelt, Ardi Roelofs, and Antje S. Meyer. 1999. A theory of lexical access in speech production. Behavioral and Brain Sciences, 22(1):1–75. find
  33. Jiwei Li, Xinlei Chen, Eduard H. Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In Proceedings of NAACL-HLT 2016, pages 681–691. Franc Marušič, Andrew Nevins, and Amanda Saksida. 2007. Last-conjunct agreement in Slovenian. In Annual Workshop on Formal Approaches… find our card
  34. Janet L. Nicol, Kenneth I. Forster, and Csaba Veres. 1997. Subject–verb agreement processes in comprehension. Journal of Memory and Language, 36(4):569–587. find
  35. Joakim Nivre, Laura Rimell, Ryan McDonald, and Carlos Gomez-Rodriguez. 2010. Evaluation of dependency parsers on unbounded dependencies. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 833–841. Association for Computational Linguistics. find our card
  36. Laura Rimell, Stephen Clark, and Mark Steedman. 2009. Unbounded dependency recovery for parser evaluation. In Proceedings of EMNLP, pages 813–821. find our card
  37. Paul Rodriguez, Janet Wiles, and Jeffrey L. Elman. 1999. A recurrent neural network that learns to count. Connection Science, 11(1):5–40. find
  38. Paul Rodriguez. 2001. Simple recurrent networks learn context-free and context-sensitive languages by counting. Neural Computation, 13(9):2093–2118. find
  39. Douglas L. T. Rohde and David C. Plaut. 1999. Language acquisition in the absence of explicit negative evidence: How important is starting small? Cognition, 72(1):67– 109. find
  40. John Robert Ross. 1967. Constraints on variables in syntax. Ph.D. thesis, MIT. Carson T. Schütze. 1996. The empirical base of linguistics: Grammaticality judgments and linguistic methodology. Chicago, IL: University of Chicago Press. find
  41. Richard Socher. 2014. Recursive Deep Learning for Natural Language Processing and Computer Vision. Ph.D. thesis, Stanford University. find
  42. Adrian Staub. 2009. On the interpretation of the number attraction effect: Response time evidence. Journal of Memory and Language, 60(2):308–327. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. LSTM neural networks for language modeling. In INTERSPEECH. find
  43. Darren Tanner, Janet Nicol, and Laurel Brehm. 2014. The time-course of feature interference in agreement comprehension: Multiple mechanisms and asymmetrical attraction. Journal of Memory and Language, 76:195– 215. find
  44. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in Neural Information Processing Systems, pages 2755–2763. find
  45. Arnold Zwicky. 2005. Agreement with nearest always bad? http://itre.cis.upenn.edu/˜myl/ languagelog/archives/001846.html. *Figure 5: Activation plots for all units (see Figure 3a).* ↗