Elamur Research⌕
☰
home / papers / 2020 / On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior

On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior

Paper · 2020 · Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu et al.

Summary

This study asks which type of language model best reflects the expectations people form while reading, as seen in word-by-word reading times. The authors trained more than two dozen models from four families: LSTM, RNNG (supervised with Penn Treebank-style parses), GPT-2 Transformers trained from scratch (word-level and BPE variants) and a Kneser-Ney 5-gram. Each was trained on four nested BLLIP subsets of increasing size. Each model's surprisal was entered into regressions on human reading data from the Dundee eye-tracking corpus, Brown self-paced reading and Natural Stories, with word length and frequency as controls. Fit was scored as the per-token log-likelihood gain over a baseline (∆LogLik). Surprisal related to reading time in a near-linear way, and lower perplexity usually meant better fit. Architecture still mattered. At comparable perplexity, Transformers and n-grams did better than LSTMs and RNNGs, especially on eye movements. Syntactic generalization ability explained nothing extra once perplexity was accounted for. This implies that modeling naturalistic reading and modeling targeted syntactic tests may require different approaches.

Key points

  • Training data: BLLIP-XS (100K tokens), BLLIP-SM (5M), BLLIP-MD (14M) and BLLIP-LG (42M). Each smaller set is a subset of the larger ones, and 1–3 random seeds were trained per configuration.
  • Evaluation: surprisal of the current word and preceding words (to capture spillover) was used to predict gaze duration in Dundee and self-paced reading times in Brown and Natural Stories (SPRT). Natural Stories deliberately includes rare constructions.
  • At equal perplexity, GPT-2 and the 5-gram show higher psychometric predictive power than LSTM and RNNG. The 12-layer GPT-2 with BPE trained on larger data scores best on all three corpora.
  • Syntactic Generalization (SG) scores, taken from Hu et al. (2020), did not significantly improve ∆LogLik prediction beyond perplexity in a stepwise regression (p > 0.26 for every corpus).
  • Code and data: github.com/wilcoxeg/neural-networks-read-times.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations🎓Harvard UniversityCambridge · United States126 works · 45,130 citations

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksDevelopmental and Educational Psychology · 523 worksNeuroscience · 186 worksCognitive Neuroscience · 185 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksPsycholinguistics & language acquisition · 108 works

Cite

Cite as Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian and Roger Levy. 2020. On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior. arXiv:2006.01912.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2006.01912 · arXiv 2006.01912

@misc{wilcox2020predictive,
  title = {{On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior}},
  author = {Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian and Roger Levy},
  year = {2020},
  doi = {10.48550/arxiv.2006.01912},
  eprint = {2006.01912},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.01912}
}

References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aurnhammer, C., & Frank, S. (2019). Comparing gated and simple recurrent neural network architectures as models of human sentence processing. In Proceedings of the 41st annual conference of the cognitive science society (pp. 112– 118). find
  2. Charniak, E., Blaheta, D., Ge, N., Hall, K., Hale, J., & Johnson, M. (2000). BLLIP 1987–89 WSJ corpus release 1. Linguistic Data Consortium, Philadelphia, 36. find
  3. Dyer, C., Kuncoro, A., Ballesteros, M., & Smith, N. A. (2016). Recurrent neural network grammars. In Proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: Human language technologies, volume 1 (long papers). find
  4. Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2), 179-211. doi: 10.1207/s15516709cog1402\\ 1 Fossum, V., & Levy, R. (2012). Sequential vs. hierarchical syntactic models of human incremental sentence processing. In Proceedings of the 3rd workshop on cognitive modeling and compu… — [doi:10.1207/s15516709cog1402\](https://doi.org/10.1207/s15516709cog1402\) find
  5. Frank, S. L., & Bod, R. (2011). Insensitivity of the human sentence-processing system to hierarchical structure. Psychological science, 22(6), 829–834. find
  6. Frank, S. L., Otten, L. J., Galli, G., & Vigliocco, G. (2015). The ERP response to the amount of information conveyed by words in sentences. Brain and Language, 140, 1–11. find
  7. Futrell, R., Gibson, E., Tily, H., Blank, I., Vishnevetsky, A., Piantadosi, S. T., & Fedorenko, E. (2017). The natural stories corpus. arXiv preprint arXiv:1708.05763. find
  8. Futrell, R., Wilcox, E., Morita, T., & Levy, R. (2018). RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency.
  9. Goodkind, A., & Bicknell, K. (2018). Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th workshop on cognitive modeling and computational linguistics (CMCL 2018) (pp. 10–18).
  10. Hale, J. (2001). A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the second meeting of the north american chapter of the association for computational linguistics on language technologies (pp. 1–8).
  11. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780. doi: 10 .1162/neco.1997.9.8.1735 Hu, J., Gauthier, J., Qian, P., Wilcox, E., & Levy, R. P. (2020). A systematic assessment of syntactic generalization in neural language models. In Proceedings of t… find
  12. Kennedy, A. (2003). The Dundee Corpus [cd-rom]. Psychology Department, University of Dundee. find
  13. Levy, R. (2008). Expectation-based syntactic comprehension. Cognition, 106(3), 1126–1177. find
  14. Marcus, M. P., Marcinkiewicz, M. A., & Santorini, B. (1993). Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19, 313–330.
  15. Marvin, R., & Linzen, T. (2018, October-November). Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 1192–1202). Brussels, Belgium: Association for Computational Linguistics. Retrieved from https://www.acl… ↗
  16. Merkx, D., & Frank, S. L. (2020). Comparing transformers and rnns on predicting human sentence processing data. arXiv preprint arXiv:2005.09471. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., De-Vito, Z., . . . Lerer, A. (2017). Automatic differentiation in PyTorch. In Neural information… — [arXiv:2005.09471](https://arxiv.org/abs/2005.09471) find
  17. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. find
  18. Sennrich, R., Haddow, B., & Birch, A. (2015). Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
  19. Smith, N. J., & Levy, R. (2008). Optimal processing times in reading: a formal model and empirical investigation. In Proceedings of the 30th annual meeting of the Cognitive Science Society (pp. 595–600). find
  20. Washington, DC. Smith, N. J., & Levy, R. (2013). The effect of word predictability on reading time is logarithmic. Cognition, 128(3), 302–319. find
  21. Staub, A., & Clifton, C. (2006). Syntactic prediction in language comprehension: Evidence from either . . . or. Journal of Experimental Psychology: Learning, Memory, & Cognition, 32(2), 425–436. find
  22. Stolcke, A. (2002). SRILM – an extensible language modeling toolkit. In Seventh international conference on spoken language processing. find
  23. Stowe, L. A. (1986). Parsing wh-constructions: Evidence for on-line gap location. Language & Cognitive Processes, 1(3), 227–245. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., . . . Polosukhin, I. (2017). Attention is all you need. In I. Guyon et al. (Eds.), Advances i…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wilcox-2020-predictive-power-neural-language-models/