Elamur Research
☰
home / papers / 2018 / What do RNN Language Models Learn about Filler-Gap Dependencies?

What do RNN Language Models Learn about Filler-Gap Dependencies?

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1809.00042

Summary

The paper asks whether LSTM language models trained only on raw English text acquire filler–gap dependencies. In these dependencies a wh-word such as "who" or "what" licenses an empty syntactic position later in the sentence. The authors also ask whether the models respect island constraints that block such gaps. They probe two pre-trained LSTMs, the large Google model and the smaller Gulordava model, and use a 5-gram baseline for comparison. The test items are hand-built sentences in a 2×2 design that crosses the presence of a wh-licensor with the presence of a gap. The measure is surprisal, either at the word after the gap or summed over the embedded clause. From these values they compute a "wh-licensing interaction" and test its significance with mixed-effects regression.

Both LSTMs show licensing for gaps in subject, object and prepositional-object positions. The effect survives intervening material and assumes one gap per filler. Licensing weakens inside wh-islands, adjunct islands and complex NP islands, with some model-specific failures, but subject islands are not learned. This suggests that much of this syntax can be learned without a built-in hierarchical bias.

Key points

  • Models tested:
    • The Google model (BIG LSTM+CNN Inputs, Jozefowicz et al. 2016) used the One Billion Word Benchmark as its training corpus. It has 2 layers of 8196 units and character-CNN inputs.
    • The Gulordava model was trained on 90M tokens of English Wikipedia and has 2 layers of 650 units.
    • The baseline is a KenLM 5-gram model with modified Kneser-Ney smoothing.
  • Complex NP islands: In 7 of 8 comparisons, licensing was significantly reduced. The Google model failed on that-headed relative clauses in subject position.
  • Subject constraint: Neither model learned it. The Google model treated PPs as open to extraction everywhere, while the Gulordava model treated them as islands everywhere.
  • Model size: The Google model has about ten times the data and hidden units, yet it was not qualitatively more human-like. The Gulordava model missed wh-islands.
  • Materials: The studies were preregistered on aspredicted.org. Materials and scripts are available at osf.io/zpfxm.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

122 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 116 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1809.00042 · arXiv 1809.00042

@misc{wilcox2018what,
  title = {{What do RNN Language Models Learn about Filler-Gap Dependencies?}},
  author = {Wilcox, Ethan and Levy, Roger and Morita, Takashi and Futrell, Richard},
  year = {2018},
  doi = {10.48550/arxiv.1809.00042},
  eprint = {1809.00042},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1809.00042}
}

References

34 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ambridge, Ben and Adele E Goldberg. 2008. The island status of clausal complements: Evidence in favor of an information structure explanation. Cognitive Linguistics 19(3):357–389. find
  2. Baayen, R.H., D.J. Davidson, and D.M. Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of memory and language 59(4):390– 412. find
  3. Barr, Dale J, Roger Levy, Christoph Scheepers, and Harry J Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language 68(3):255–278. find
  4. Bartek, B., Richard L. Lewis, Shravan Vasishth, and M. R. Smith. 2011. In search of on-line locality effects in sentence comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition 37(5):1178–1198. find
  5. Chelba, Ciprian, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005 . find our card
  6. Chowdhury, Shammur Absar and Roberto Zamparelli. 2018. Rnn simulations of grammaticality judgments on long-distance dependencies. In Proceedings of the 27th International Conference on Computational Linguistics. pages 133–144. find
  7. Ellefson, Michelle R and Morten H Christiansen. 2000. Subjacency constraints without universal grammar: Evidence from artificial language learning and connectionist modeling. In Proceedings of the Annual Meeting of the Cognitive Science Society. volume 22. find
  8. Elman, Jeffrey L. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine learning 7(2-3):195–225. find
  9. Elman, J.L. 1990. Finding structure in time. Cognitive Science 14(2):179–211. find
  10. Goldberg, Yoav. 2017. Neural network methods for natural language processing. Synthesis Lectures on Human Language Technologies 10(1):1–309. find
  11. Grodner, Daniel and Edward Gibson. 2005. Consequences of the serial nature of linguistic input for sentential complexity. Cognitive Science 29(2):261–290. find
  12. Gulordava, K., P. Bojanowski, E. Grave, T. Linzen, and M. Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of NAACL. find our card
  13. Hale, John T. 2001. A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics and Language Technologies. pages 1–8. find our card
  14. Heafield, Kenneth, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. Sofia, Bulgaria. find our card
  15. Hochreiter, Sepp and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8):1735–1780. find
  16. Hofmeister, Philip and Ivan A Sag. 2010. Cognitive constraints and island effects. Language 86(2):366. find
  17. Huang, Cheng-Teh James. 1982. Logical relations in Chinese and the theory of grammar. Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, MA. find
  18. Jozefowicz, Rafal, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv 1602.02410. find
  19. King, Jonathan and Marcel Adam Just. 1991. Individual differences in syntactic processing: The role of working memory. Journal of memory and language 30(5):580. find
  20. Lau, Jey Han, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: a probabilistic view of linguistic knowledge. Cognitive Science 41(5):1202–1241. find
  21. Levy, Roger. 2008. Expectation-based syntactic comprehension. Cognition 106(3):1126–1177. find
  22. Linzen, Tal, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics 4:521–535. find our card
  23. Masson, Michael E. J. and Geoffrey R. Loftus. 2003. Using confidence intervals for graphically based data interpretation. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale 57(3):203. find
  24. McCoy, R. Thomas, Robert Frank, and Tal Linzen. 2018. Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks. arXiv preprint arXiv:1802.09091 . — [arXiv:1802.09091](https://arxiv.org/abs/1802.09091) find
  25. Pearl, Lisa and Jon Sprouse. 2013. Syntactic islands and learning biases: Combining experimental syntax and computational modeling to investigate the language acquisition problem. Language Acquisition 20(1):23–68. find
  26. Phillips, Colin. 2006. The real-time status of island phenomena. Language pages 795–823. find
  27. Phillips, Colin. 2013. On the nature of island constraints II: Language learning and innateness. Experimental syntax and island effects pages 132–157. find
  28. Ross, John Robert. 1967. Constraints on variables in syntax. Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, MA. find
  29. Smith, Nathaniel J. and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition 128(3):302–319. find
  30. Sprouse, Jon and Norbert Hornstein. 2014. Experimental syntax and island effects. Cambridge University Press. find
  31. Stowe, Laurie A. 1986. Parsing wh-constructions: Evidence for on-line gap location. Language and cognitive processes 1(3):227–245. find
  32. Sutskever, Ilya, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. pages 3104–3112. find
  33. Traxler, Matthew J and Martin J Pickering. 1996. Plausibility and the processing of unbounded dependencies: An eye-tracking study. Journal of Memory and Language 35(3):454–475. find
  34. Vinyals, Oriol, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in Neural Information Processing Systems. pages 2773–2781. find