Elamur Research⌕
☰
home / papers / 2018 / RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency

RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency

Summary

The paper asks which aspects of grammar LSTM language models pick up when trained only to predict the next word. The authors treat the networks the way psycholinguists treat human participants. They build controlled sets of sentences and use per-word surprisal as a stand-in for human reading time, analysing it with linear mixed-effects regression. All analyses were preregistered.

Three models are tested: two public English LSTMs (JRNN and GRNN) and a new character-level Japanese LSTM (JPRNN). The study probes two kinds of knowledge:

  • Syntactic state: garden-path sentences, center-embedding, and expectations for obligatory upcoming material.
  • Grammatical dependency: reflexive pronoun binding and licensing of negative polarity items (NPIs).

The models clearly track incremental syntactic structure and show human-like garden-path effects. They also maintain expectations for required continuations, although these weaken as the intervening material grows longer or more complex. However, none of the models learns the correct structural conditions for reflexives or NPIs, in English or Japanese. The authors conclude that syntax can emerge from string input alone but does not reliably generalize across constructions.

Key points

  • Models tested:
    • JRNN (Jozefowicz et al. 2016): trained on the One Billion Word Benchmark, with 2 layers of 8196 units and CNN character inputs.
    • GRNN (Gulordava et al. 2018): trained on 90M tokens of English Wikipedia, with 2 layers of 650 units.
    • JPRNN: character-based, 650 units, trained on 800k of 900k Japanese Wikipedia paragraphs (the rest held out for validation), reaching validation perplexity 12.67.
  • Garden paths: In 29 Main Verb/Reduced Relative items, surprisal at the disambiguating verb is highest in the condition that pairs a reduced relative clause with an ambiguous verb form (interaction p < 0.001 in both English models).
  • Partial verb-form knowledge: An unambiguous participle lowers surprisal but does not remove the reduced-RC penalty. The models apparently treat the participle as only a noisy cue to a relative clause.
  • Japanese NPI shika: The model learns that shika predicts a later negated verb. It fails, however, to restrict this expectation to the structurally correct clause and wrongly carries it over into the matrix clause.
  • Resources: Scripts, materials, and results are available at github.com/Futrell/rnn_psycholinguistic_subjects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Psycholinguistics & language acquisition

Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States125 works · 60 031 citations🎓Harvard UniversityCambridge · United States64 works · 35 670 citations🎓University of California, IrvineIrvine · United States22 works · 35 120 citations🎓Kyoto UniversityKyoto · Japan21 works · 7 437 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1809.01329 · arXiv 1809.01329

@misc{futrell2018rnns,
  title = {{RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency}},
  author = {Futrell, Richard and Wilcox, Ethan and Morita, Takashi and Levy, Roger},
  year = {2018},
  doi = {10.48550/arxiv.1809.01329},
  eprint = {1809.01329},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1809.01329}
}

References

58 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hiroshi Aoyagi and Toru Ishii. 1994. On agreement-inducing vs. non-agreement-inducing NPIs. In Proceedings of the North East Linguistic Society 24, pages 1 – 15. GLSA Publications. find
  2. R.H. Baayen, D.J. Davidson, and D.M. Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4):390–412. find
  3. Dale J Barr, Roger Levy, Christoph Scheepers, and Harry J Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3):255–278. find
  4. Yonatan Belinkov, Lluı́s Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2018. Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks. arXiv preprint arXiv:1801.07772. find our card
  5. Jean-Philippe Bernardy and Shalom Lappin. 2017. Using deep neural networks to learn syntactic agreement. Linguistic Issues in Language Technology, 15:1–15. find
  6. Thomas G. Bever. 1970. The cognitive basis for linguistic structures. In J. R. Hayes, editor, Cognition and the Development of Language. Wiley, New York. find
  7. Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. Deep RNNs encode soft hierarchical syntax. arXiv preprint arXiv:1805.04218. find
  8. Kathryn Bock and Carol A Miller. 1991. Broken agreement. Cognitive Psychology, 23(1):45–93. find
  9. Bureau of Labor Statistics. 2017. Labor force statistics from the current population survey. find
  10. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  11. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005. find our card
  12. N. Chomsky. 1981. Lectures on Government and Binding. Foris Publications, Dordrecht, The Netherlands. find
  13. Morten H. Christiansen and Nick Chater. 1999. Toward a connectionist model of recursion in human linguistic performance. Cognitive Science, 23(2):157–205. find
  14. Charles Clifton Jr., Matthew J. Traxler, Mohamed Taha Mohamed, Rihana S. Williams, Robin K. Morris, and Keith Rayner. 2003. The use of thematic role information in parsing: Syntactic processing autonomy revisited. Journal of Memory and Language, 49:317–334. find
  15. Brian Dillon, Alan Mishler, Shayne Sloggett, and Colin Phillips. 2013. Contrasting intrusion profiles for agreement and anaphora: Experimental and modeling evidence. Journal of Memory and Language, 69(2):85–103. find
  16. J.L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. find
  17. Emile Enguehard, Yoav Goldberg, and Tal Linzen. 2017. Exploring the syntactic abilities of RNNs with multi-task learning. arXiv preprint arXiv:1706.03542. find
  18. Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M Bender. 2017. Towards linguistically generalizable NLP systems: A workshop and shared task. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems, pages 1–10. find
  19. Fernanda Ferreira and Charles Clifton. 1986. The independence of syntactic processing. Journal of Memory and Language, 25(3):348–368. find
  20. Stefan L. Frank and Rens Bod. 2011. Insensitivity of the human sentence-processing system to hierarchical structure. Psychological Science, 22(6):829– 834. find
  21. Stefan L. Frank, Thijs Trompenaars, Richard L. Lewis, and Shravan Vasishth. 2016. Cross-linguistic differences in processing double-embedded relative clauses: Working-memory constraints or language statistics? Cognitive Science, 40:554–578. find
  22. Richard Futrell and Roger Levy. 2017. Noisy-context surprisal as a human sentence processing cost model. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 688–698, Valencia, Spain. find
  23. Edward Gibson and James Thomas. 1999. Memory limitations and structural forgetting: The perception of complex ungrammatical sentences as grammatical. Language and Cognitive Processes, 14(3):225– 248. find
  24. Yoav Goldberg. 2017. Neural network methods for natural language processing. Synthesis Lectures on Human Language Technologies, 10(1):1–309. find
  25. K. Gulordava, P. Bojanowski, E. Grave, T. Linzen, and M. Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of NAACL. find our card
  26. John T. Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics and Language Technologies, pages 1–8. find our card
  27. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv, 1602.02410. find
  28. Ákos Kádár, Grzegorz Chrupała, and Afra Alishahi. 2017. Representation of linguistic form and function in recurrent neural networks. Computational Linguistics, 43(4):761–780. find
  29. Kiyoko Kataoka. 2006. Nihongo hitei-bun no koozoo: kakimaze-bun to hitei-koou hyoogen. Kuroshio, Tokyo. find
  30. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, pages 2741–2749. AAAI Press. find our card
  31. Brenden M Lake and Marco Baroni. 2017. Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks. arXiv preprint arXiv:1711.00350. find
  32. Ellen Lau, Clare Stroud, Silke Plesch, and Colin Phillips. 2006. The role of structural prediction in rapid syntactic analysis. Brain & Language, 98:74– 88. find
  33. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  34. Roger Levy, Evelina Fedorenko, Mara Breen, and Ted Gibson. 2012. The processing of extraposed structures in English. Cognition, 122(1):12–36. find
  35. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521– 535. find our card
  36. Maryellen C. MacDonald and Morten H. Christiansen. 2002. Reassessing working memory: Comment on Just and Carpenter (1992) and Waters and Caplan (1996). Psychological Review, 109(1):35–54. find
  37. Maryellen C. MacDonald, Neal J. Pearlmutter, and Mark S. Seidenberg. 1994. The lexical nature of syntactic ambiguity resolution. Psychological Review, 101(4):676. find
  38. Michael EJ Masson and Geoffrey R Loftus. 2003. Using confidence intervals for graphically based data interpretation. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale, 57(3):203. find
  39. G.A. Miller and N. Chomsky. 1963. Finitary models of language users. Handbook of Mathematical Psychology, 2:419–491. find
  40. Masatake Muraki. 1978. The sika nai construction and predicate restructuring. In John Hinds and Irwin Howard, editors, Problems in Japanese syntax and semantics, pages 155 – 177. Kaitakusha, Tokyo. find
  41. Carl Pollard and Ivan A. Sag. 1994. Head-Driven Phrase Structure Grammar. Center for the Study of Language and Information, Stanford, CA. find
  42. Tanya Reinhart. 1981. Definite NP anaphora and c-command domains. Linguistic Inquiry, 12(4):605– 635. find
  43. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301. find our card
  44. Marten van Schijndel and Tal Linzen. 2018. Modeling garden path effects without explicit hierarchical syntax. In Proceedings of the 40th Annual Meeting of the Cognitive Science Society. find
  45. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526– 1534. find our card
  46. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319. find
  47. Adrian Staub and Charles Clifton. 2006. Syntactic prediction in language comprehension: Evidence from either . . . or. Journal of Experimental Psychology: Learning, Memory, & Cognition, 32(2):425–436. find
  48. Patrick Sturt. 2003. The time-course of the application of binding constraints in reference resolution. Journal of Memory and Language, 48:542–562. find
  49. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. find
  50. Hidekazu Tanaka. 1997. Invisible movement in sika-nai and the linear crossing constraint. Journal of East Asian Linguistics, 6(2):143–188. find
  51. John C. Trueswell, Michael K. Tanenhaus, and S. Garnsey. 1994. Semantic influences on parsing: Use of thematic role information in syntactic ambiguity resolution. Journal of Memory and Language, 33:285– 318. find
  52. Shravan Vasishth, Sven Brüssow, Richard L Lewis, and Heiner Drenhaus. 2008. Processing polarity: How the ungrammatical intrudes on the grammatical. Cognitive Science, 32(4):685–712. find
  53. Shravan Vasishth, Katja Suckow, Richard L Lewis, and Sabine Kern. 2010. Short-term forgetting in sentence comprehension: Crosslinguistic evidence from verb-final structures. Language and Cognitive Processes, 25(4):533–567. find
  54. Noah Weber, Leena Shekhar, and Niranjan Balasubramanian. 2018. The fine line between linguistic generalization and failure in seq2seq-attention models. In Workshop on New Forms of Generalization in Deep Learning and NLP (NAACL 2018). find
  55. Ethan G. Wilcox, Roger P. Levy, Takashi Morita, and Richard Futrell. 2018. What do RNNs learn about filler–gap dependencies? In Proceedings of BlackboxNLP 2018. find
  56. Adina Williams, Andrew Drozdov, and Samuel R Bowman. 2018. Do latent tree learning models identify meaningful structure in sentences? Transactions of the Association for Computational Linguistics, 6:253–267. find
  57. Ming Xiang, Brian Dillon, and Colin Phillips. 2009. Illusory licensing effects across dependency types: ERP evidence. Brain & Language, 108(1):40–55. find
  58. Victor H. Yngve. 1960. A model and an hypothesis for language structure. Proceedings of the American Philosophical Society, 104(5):444–466. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/futrell-2018-rnns-psycholinguistic-subjects-syntactic-state/