Elamur Research⌕
☰
home / papers / 2018 / LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better

LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better

Paper · 2018 · Adhiguna Kuncoro, Chris Dyer, John Hale et al.

Summary

Linzen et al. (2016) used subject–verb number agreement as a probe of syntactic competence. The hard cases contain intervening "attractor" nouns whose number differs from the subject. On that basis they argued that LSTM language models fail to capture long-range syntactic dependencies. Kuncoro et al. rerun this test on the same parsed Wikipedia corpus and split, training word-level LSTM language models of several hidden sizes. A model counts as correct when it assigns higher probability to the right verb form.

Larger LSTMs handle agreement well, so the earlier pessimistic result largely reflects limited capacity. A strong character-level LSTM, however, does markedly worse.

The authors then compare models with explicit syntax. Recurrent neural network grammars (RNNGs) outperform sequential LSTMs when there are several attractors. This advantage comes from RNNGs' explicit composition operator, not from merely having predicted syntactic annotations.

Finally, they add left-corner and bottom-up generation orders to RNNGs. The original top-down order performs best on the hardest cases, although all three variants reach similar perplexities. The results support a hierarchical inductive bias and show that perplexity can hide differences in grammatical generalization.

Key points

  • Data: the Linzen et al. (2016) dataset, with 141,948 training sentences and 1,211,080 test sentences. About 94.7% of test cases have no attractors. Words outside the top 10,000 types are replaced by POS tags, and models are implemented in DyNet.

  • Capacity matters: with two attractors, the reported 50-unit LSTM error was about 50%. The authors' own models reach 15.7% at H=50 and 5.7% at H=350. With four attractors, the authors' H=350 model has 13.8% error.

  • The character LSTM of Melis et al. (2018) has 1,800 hidden units and 10M parameters. It reaches 27.8% error at four attractors, consistent with a sequential recency bias.

  • Mean error at four attractors over 10 seeds:

    ModelError
    Best sequential LSTM14.1%
    Top-down RNNG8.9%
    Bottom-up RNNG9.7%
    Left-corner RNNG9.9%

    The top-down advantage at four attractors is significant at p < 0.05 under an approximate randomization test.

  • Syntactic language models without explicit composition give no agreement benefit over plain sequential LSTMs. This holds even though such models can be good predictors of phrase structure.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cornell UniversityIthaca · United States89 works · 58,422 citations🎓University of OxfordOxford · United Kingdom52 works · 154,829 citations🏢Google DeepMindLondon · United Kingdom35 works · 75,886 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Syntax & grammar formalisms · 161 works

Cite

Cite as Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark and Phil Blunsom. 2018. LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1132

@inproceedings{kuncoro2018lstms,
  title = {{LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better}},
  author = {Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark and Phil Blunsom},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1132},
  url = {https://doi.org/10.18653/v1/p18-1132}
}

References

36 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Abney and Mark Johnson. 1991. Memory requirements and local ambiguities for parsing strategies. Journal of Psycholinguistic Research . find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR.
  3. Eugene Charniak. 1997. Statistical techniques for natural language parsing. AI Magazine . find
  4. Eugene Charniak. 2010. Top-down nearly-context-sensitive parsing. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Cambridge, MA, pages 674–683. http://www.aclweb.org/anthology/D10-1066. ↗
  5. Ciprian Chelba and Frederick Jelinek. 2000. Structured language modeling. Computer Speech and Language 14(4). find
  6. Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proc. of EMNLP. find
  7. Philip N. Johnson-Laird. 1983. Mental Models. Harvard University Press. find
  8. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. find
  9. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network grammars learn about syntax? In Proc. of EACL. find
  10. Gaël Le Godais, Tal Linzen, and Emmanuel Dupoux. 2017. Comparing character-level neural language models using a lexical decision task. In Proc. of EACL. find
  11. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics .
  12. Christopher D. Manning and Bob Carpenter. 1997. Probabilistic parsing using left corner language models. In Proc. of IWPT. find
  13. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proc. of NAACL. find
  14. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics .
  15. Ahmad Emami and Frederick Jelinek. 2005. A neural syntactic language model. Machine Learning 60:195–227. find
  16. William Marslen-Wilson. 1973. Linguistic structure and speech shadowing at very short latencies. Nature 244:522–523. find
  17. Daniel Fried, Mitchell Stern, and Dan Klein. 2017. Improving neural parsing by disentangling model combination and reranking effects. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Vancouver, Canada, pages 161–166. find
  18. Gabor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In Proc. of ICLR. find
  19. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proc. of NAACL.
  20. John T Hale. 2014. Automaton theories of human sentence comprehension. CSLI Publications. find
  21. James Henderson. 2003. Inducing history representations for broad coverage statistical parsing. In Proc. of NAACL. find
  22. James Henderson. 2004. Discriminative training of a neural network statistical parser. In Proc. of ACL. find
  23. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation . find
  24. Marcus Hutter. 2012. The human knowledge compression contest . find
  25. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proc. of EMNLP.
  26. Matthew J. Nelson, Imen El Karoui, Kristof Giber, Xiaofang Yang, Laurent Cohen, Hilda Koopman, Sydney S. Cash, Lionel Naccache, John T. Hale, Christophe Pallier, and Stanislas Dehaene. 2017. Neurophysiological dynamics of phrase-structure building during sentence processing. Proceedings of the Natio… find
  27. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Pau…
  28. Stephen Pulman. 1986. Grammars, parsers, and memory limitations. Language and Cognitive Processes . find
  29. Philip Resnik. 1992. Left-corner parsing and psychological plausibility. In Proc. of COLING. find
  30. Laura Rimell, Stephen Clark, and Mark Steedman. 2009. Unbounded dependency recovery for parser evaluation. In Proc. of EMNLP.
  31. William Schuler, Samir AbdelRahman, Tim Miller, and Lane Schwartz. 2010. Broad-coverage parsing using human-like memory constraints 36(1):1–30. find
  32. Abigail See, Peter Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proc. of ACL. find
  33. Rico Sennrich. 2017. How grammatical is character-level neural machine translation? assessing mt quality with contrastive translation pairs. In Proc. of EACL.
  34. Michael Tanenhaus, Michael Spivey-Knowlton, Kathleen Eberhard, and Julie Sedivy. 1995. Integration of visual and linguistic information in spoken language comprehension. Science 268:1632–1634. find
  35. Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In Proc. of COLING. find
  36. Dani Yogatama, Yishu Miao, Gabor Melis, Wang Ling, Adhiguna Kuncoro, Chris Dyer, and Phil Blunsom. 2018. Memory architectures in recurrent neural network language models. In Proc. of ICLR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kuncoro-2018-lstms-can-learn-syntax-sensitive/