Elamur Research
☰
home / papers / 2018 / Predictive power of word surprisal for reading times is a linear function of language model quality

Predictive power of word surprisal for reading times is a linear function of language model quality

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-0102.pdf

Summary

Surprisal theory links reading difficulty to a word's negative log probability in context. Researchers usually estimate these probabilities with computational language models, which predict upcoming words less well than humans do. This raises the question of whether weaker models distort conclusions about surprisal. Goodkind and Bicknell trained seven language models of varying quality on the Google One Billion Word Benchmark: n-gram models, an LSTM, and two LSTM/5-gram interpolations. They used each model's surprisal estimates to predict first-pass gaze durations in the English Dundee Corpus, fitting generalized additive mixed-effects models with both linear and nonlinear surprisal terms. As perplexity dropped, the models' gain in fit to reading times grew along a tight linear trend. Across all model types, the estimated size of the surprisal effect stayed roughly the same, and its shape looked approximately linear. These results suggest that surprisal findings based on weaker language models are not systematically biased. They also suggest that better language models yield better cognitive predictions.

Key points

  • Models compared: seven in total.
    • Four kenlm n-gram models (2-gram to 5-gram) with Modified Kneser-Ney smoothing; unigram counts served only as a frequency control.
    • One word-level LSTM.
    • Two LSTM+5-gram interpolations: one with perplexity-optimal weights of 0.71/0.29, and one with equal weights.
  • Training data: the "1b corpus" (about 0.8 billion words, vocabulary of roughly 800,000).
  • Perplexity on Dundee (all tokens): from 73.39 for the optimal interpolation to 290.88 for the bigram model; the LSTM scored 113.27 and the 5-gram 168.98.
  • Reading data: Dundee Corpus, 10 readers of editorials from The Independent. After exclusions, 289,726 gaze durations remained, and 175 out-of-vocabulary tokens were dropped.
  • Nonlinear GAMM fit: the gain in log-likelihood ranked models the same way as perplexity, from 297 for the optimal interpolation to 166 for the bigram model. The surprisal curves stayed close to linear, extending the earlier findings of Smith & Levy (2013).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

114 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-0102

@inproceedings{goodkind2018predictive,
  title = {{Predictive power of word surprisal for reading times is a linear function of language model quality}},
  author = {Goodkind, Adam and Bicknell, Klinton},
  booktitle = {Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018)},
  year = {2018},
  doi = {10.18653/v1/w18-0102},
  url = {https://doi.org/10.18653/v1/w18-0102}
}

References

19 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Monsalve, I. F., Frank, S. L., & Vigliocco, G. (2012). Lexical surprisal as a general predictor of reading time. In Proceedings of the 13th conference of the european chapter of the association for computational linguistics (pp. 398–408). find
  2. Bicknell, K., & Levy, R. (2010). A rational model of eye movement control in reading. In J. Hajivc, S. Carberry, S. Clark, & J. Nivre (Eds.), Proceedings of the 48th annual meeting of the association for computational linguistics (acl) (pp. 1168– 1178). Uppsala, Sweden: Association for Computational… find
  3. R Core Team. (2013). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved from http:// www.R-project.org/Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2013). One billion word benchmark for measuring pro… find our card
  4. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423. Retrieved from http://dx.doi.org/10.1002/ j.1538-7305.1948.tb01338.x doi: 10.1002/j.1538-7305.1948.tb01338.x — [doi:10.1002/j.1538-7305.1948.tb01338.x](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x) find
  5. Demberg, V., & Keller, F. (2008). Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2), 193– 210. find
  6. Smith, N., & Levy, R. (2011). Cloze but no cigar: The complex relationship between cloze, corpus, and subjective probabilities in language processing. In Proceedings of the cognitive science society (Vol. 33). find
  7. Fossum, V., & Levy, R. (2012). Sequential vs. hierarchical syntactic models of human incremental sentence processing. In Proceedings of the 3rd workshop on cognitive modeling and computational linguistics (pp. 61–69). find
  8. Smith, N., & Levy, R. (2013). The effect of word predictability on reading time is logarithmic. Cognition, 128(3), 302–319. find
  9. Frank, S. L., & Bod, R. (2011). Insensitivity of the human sentence-processing system to hierarchical structure. Psychological science, 22(6), 829–834. find
  10. Stolcke, A., Zheng, J., Wang, W., & Abrash, V. (2011). Srilm at sixteen: Update and outlook. In Proceedings of ieee automatic speech recognition and understanding workshop (Vol. 5). find
  11. Frank, S. L., Monsalve, I. F., Thompson, R. L., & Vigliocco, G. (2013). Reading time data for evaluating broad-coverage models of english sentence processing. Behavior Research Methods, 45(4), 1182–1190. find
  12. Taylor, W. L. (1953). ” Cloze procedure”: a new tool for measuring readability. Journalism quarterly. find
  13. Hale, J. (2001). A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the second meeting of the north american chapter of the association for computational linguistics on language technologies (pp. 1–8). find our card
  14. Wood, S. N. (2004). Stable and efficient multiple smoothing parameter estimation for generalized additive models. Journal of the American Statistical Association, 99(467), 673-686. find
  15. Wood, S. N. (2017). Generalized additive models: An introduction with R (2nd ed.). Chapman and Hall/CRC. find
  16. Heafield, K., Pouzyrevsky, I., Clark, J. H., & Koehn, P. (2013, August). Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st annual meeting of the association for computational linguistics (pp. 690–696). Sofia, Bulgaria. Retrieved from https://kheafield.com/papers/ edi… ↗
  17. Kennedy, A., Hill, R., & Pynte, J. (2003). The dundee corpus. In Proceedings of the 12th european conference on eye movement. find
  18. Levy, R. (2008). Expectation-based syntactic comprehension. Cognition, 106(3), 1126–1177. find
  19. Mitchell, J., Lapata, M., Demberg, V., & Keller, F. (2010). Syntactic and semantic factors in processing difficulty: An integrated measure. In Proceedings of the 48th annual meeting of the association for computational linguistics (pp. 196–206). find