Elamur Research
☰
home / papers / 2019 / Neural language models as psycholinguistic subjects: Representations of syntactic state

Neural language models as psycholinguistic subjects: Representations of syntactic state

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-1004.pdf

Summary

Futrell and colleagues test whether neural language models track incremental syntactic state. This is the information a stack-based parser would hold, such as knowing that an open subordinate clause still needs a main clause. They borrow methods from psycholinguistics and treat per-word surprisal as a stand-in for human reading time. Surprisal is measured on hand-built sentence sets that target subordinate clauses and two garden-path ambiguities: NP/Z and main verb/reduced relative (MV/RR). Condition contrasts are analysed with linear mixed-effects regression.

Four English models are compared. JRNN (One Billion Word Benchmark) and GRNN (Wikipedia) are large LSTMs. An RNNG and a matched TinyLSTM are both trained on about 1 million Penn Treebank tokens. All models show basic evidence of representing syntactic state. Only the large-data models, and in some tests the RNNG, use subtler lexical cues such as verb transitivity or participle morphology. Unlike humans, the models' garden-path effects shrink rather than grow as the ambiguous region lengthens. The work offers a reusable targeted-evaluation framework and shows that both training-data scale and explicit syntactic structure shape what models learn.

Key points

  • Models and training data:

    • JRNN: LSTM, about 800M tokens.
    • GRNN: LSTM, about 90M tokens.
    • RNNG: about 1M Penn Treebank tokens; surprisal estimated with word-synchronous beam search (action beam 100, word beam 10).
    • TinyLSTM: about 1M Penn Treebank tokens.

  • Subordinate clauses: the authors define two measures.

    • Matrix licensing effect: lower surprisal for a main clause after a subordinator.
    • No-matrix penalty: higher surprisal when the main clause is missing.
    • Only GRNN and the RNNG showed the penalty.

  • MV/RR garden path (29 items): reducing the relative clause adds the following surprisal at the disambiguating verb:

    ModelAdded surprisal
    TinyLSTM0.3 bits
    JRNN1.9 bits
    RNNG2.8 bits
    GRNN3.6 bits

    An unambiguous participle (e.g., "given") lowers but does not remove this effect, so models treat morphology as only a noisy cue.

  • Fine-grained cues: TinyLSTM failed every fine-grained test. The RNNG missed the verb-transitivity effect in NP/Z sentences.

  • Released materials: items, results and analysis scripts are at github.com/langprocgroup/nn_syntactic_state.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Treebanks

Cited by

147 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 140 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1004 · arXiv 1903.03260

@inproceedings{futrell2019neural,
  title = {{Neural language models as psycholinguistic subjects: Representations of syntactic state}},
  author = {Futrell, Richard and Wilcox, Ethan and Morita, Takashi and Qian, Peng and Ballesteros, Miguel and Levy, Roger},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1004},
  eprint = {1903.03260},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1004}
}

References

4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Haraald Baayen, D.J. Davidson, and D.M. Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4):390–412. find
  2. Dale J. Barr, Roger P. Levy, Christoph Scheepers, and Harry J Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3):255–278. find
  3. Jean-Philippe Bernardy and Shalom Lappin. 2017. Using deep neural networks to learn syntactic agreement. Linguistic Issues in Language Technology, 15:1–15. find
  4. Thomas G. Bever. 1970. The cognitive basis for linguistic structures. In J. R. Hayes, editor, Cognition and the Development of Language. Wiley, New York. find