Elamur Research⌕
☰
home / papers / 2025 / The Impact of Token Granularity on the Predictive Power of Language Model Surprisal

The Impact of Token Granularity on the Predictive Power of Language Model Surprisal

Summary

Cognitive modeling often uses word-level surprisal from language models to predict how hard words are for readers, but the effect of subword token granularity has rarely been examined. Oh and Schuler varied granularity by training unigram language model (ULM) tokenizers on English Wiki-40B with vocabularies from 256 to 128,000 entries. They then trained Mamba-2 state space LMs in three sizes for each vocabulary. Surprisal was tested against naturalistic reading times from several corpora, both before and after training, and against garden-path stimuli from Huang et al. (2024). On reading times, granularity mattered a great deal. Vocabularies around 8,000, finer than the roughly 50,000 used by common LMs, gave the most predictive surprisal. The authors attribute this to word length and frequency information built into tokenization. On garden-path sentences, coarser, more word-like tokens generally produced higher surprisal at critical regions. Tokenization is therefore a modeling choice that cognitive researchers should tune deliberately, and the best setting may differ between broad-coverage reading and syntactic ambiguity effects.

Key points

  • Tokenizers: ULM tokenizers were trained on 1,000,000 English Wiki-40B articles. Eleven vocabulary sizes were tested (256 to 128,000), built on 158 base characters at 99.95% character coverage.
  • Models: Mamba-2 LMs came in three sizes, Small (about 2.6M), Medium (about 19.8M) and Large (about 88M) non-embedding parameters. This architecture was chosen because finer tokenization produces very long sequences, which suits Transformers poorly.
  • Untrained models: Even before any training, granularity alone strongly shaped how well surprisal fit reading times, with about 4,000 tokens fitting best. Smaller trained models kept this bias and ended up more predictive than larger ones.
  • Garden-path effects: Small LMs with larger vocabularies showed bigger effects on the MV/RR construction, and larger vocabularies generally widened surprisal differences at critical words. However, no granularity setting closed the large gap to the size of human garden-path effects.
  • Data and limitations: The reading-time measures were drawn from five corpora: Luke & Christianson 2018, Kennedy et al. 2003, Cop et al. 2017, Futrell et al. 2021, and Smith & Levy 2013. The study covers English only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Language models & pretraining · Psycholinguistics & language acquisition

Institutions

🎓New York UniversityNew York · United States106 works · 82 819 citations🎓The Ohio State UniversityColumbus · United States46 works · 16 163 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.209

@inproceedings{oh2025impact,
  title = {{The Impact of Token Granularity on the Predictive Power of Language Model Surprisal}},
  author = {Oh, Byung-Doh and Schuler, William},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.209},
  url = {https://doi.org/10.18653/v1/2025.acl-long.209}
}

References

43 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Suhas Arehalli, Brian Dillon, and Tal Linzen. 2022. Syntactic surprisal from neural models predicts, but underestimates, human processing difficulty from syntactic ambiguities. In Proceedings of the 26th Conference on Computational Natural Language Learning, pages 301–313. find
  2. Catherine Arnett and Benjamin Bergen. 2025. Why do language models perform worse for morphologically complex languages? In Proceedings of the 31st International Conference on Computational Linguistics, pages 6607–6623. find
  3. Dale J. Barr, Roger Levy, Christoph Scheepers, and Harry J. Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68:255–278. find
  4. Jason J. S. Barton, Hashim M. Hanif, Laura Eklinder Björnstöm, and Charlotte Hills. 2014. The word-length effect in reading: A review. Cognitive Neuropsychology, 31(5–6):378–412. find
  5. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1):1– 48. find our card
  6. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across… find
  7. Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617–4624. find our card
  8. Agatha Christie. 1920. The Mysterious Affair at Styles. John Lane. Retrieved from Project Gutenberg. find
  9. Uschi Cop, Nicolas Dirix, Denis Drieghe, and Wouter Duyck. 2017. Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading. Behavior Research Methods, 49(2):602–615. find
  10. Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pages 10041–10071. find
  11. Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko. 2021. The Natural Stories corpus: A reading-time corpus of English texts containing rare syntactic constructions. Language Resources and Evaluation, 55:63–77. find
  12. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  13. Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira, and Ryan Cotterell. 2024. On the proper treatment of tokenization in psycholinguistics. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18556–18572. find our card
  14. Steven G. Luke and Kiel Christianson. 2018. The Provo Corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50(2):826– 833. find
  15. Aaron Gokaslan and Vanya Cohen. 2019. OpenWeb-Text Corpus. http://Skylion007.github.io/ OpenWebTextCorpus. ↗
  16. Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. 2020. Wiki-40B: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2440–2452. find
  17. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics on Language Technologies, pages 1–8. find our card
  18. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 690–696. find our card
  19. Kuan-Jung Huang, Suhas Arehalli, Mari Kugemoto, Christian Muxica, Grusha Prasad, Brian Dillon, and Tal Linzen. 2024. Large-scale benchmark yields no evidence that language model surprisal explains syntactic disambiguation difficulty. Journal of Memory and Language, 137:104510. find
  20. Marcel Adam Just and Patricia A. Carpenter. 1980. A theory of reading: From eye fixations to comprehension. Psychological Review, 87(4):329–354. find
  21. Alan Kennedy, Robin Hill, and Joël Pynte. 2003. The Dundee Corpus. In Proceedings of the 12th European Conference on Eye Movement. find
  22. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 66–75. find our card
  23. Henry Kučera and W. Nelson Francis. 1967. Computational analysis of present-day American English. Brown University Press, Providence, RI. find
  24. Sander Land and Max Bartolo. 2024. Fishing for Magikarp: Automatically detecting under-trained tokens in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11631–11646. find
  25. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  26. Danny Merkx and Stefan L. Frank. 2021. Human sentence processing: Recurrence or attention? In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 12–22. find
  27. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26, pages 3111–3119. find our card
  28. Sathvik Nair and Philip Resnik. 2023. Words, subwords, and morphemes: What really matters in the surprisal-reading time relationship? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11251–11260. find
  29. Byung-Doh Oh, Christian Clark, and William Schuler. 2021. Surprisal estimators for human reading times need character models. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lang… find
  30. Byung-Doh Oh and William Schuler. 2023a. Transformer-based language model surprisal predicts human reading times best with about two billion training tokens. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1915–1921. find
  31. Byung-Doh Oh and William Schuler. 2023b. Why does surprisal from larger Transformer-based language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:336–350. find our card
  32. Byung-Doh Oh and William Schuler. 2024. Leading whitespaces of language models’ subword vocabulary pose a confound for calculating word probabilities. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3464–3472. find
  33. Tiago Pimentel and Clara Meister. 2024. How to compute the probability of a word. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18358–18375. find
  34. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Technical Report. find
  35. Keith Rayner, Marcia Carlson, and Lyn Frazier. 1983. The interaction of syntax and semantics during sentence processing: Eye movements in the analysis of semantically biased sentences. Journal of Verbal Learning and Verbal Behavior, 22(3):358–374. find
  36. Ethan Gotlieb Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023b. Testing the predictions of surprisal theory in 11 languages. Transactions of the Association for Computational Linguistics, 11:1451–1470. find our card
  37. Craig W. Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. Tokenization is more than compression. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 678–702. find our card
  38. Weijie Xu, Jason Chon, Tianran Liu, and Richard Futrell. 2023. The linearity of the effect of surprisal on reading times across languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15711–15721. find
  39. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725. find our card
  40. Cory Shain. 2024. Word frequency and predictability dissociate in naturalistic reading. Open Mind, 8:177– 201. find
  41. Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. Large-scale evidence for logarithmic effects of word predictability on reading time. Proceedings of the National Academy of Sciences, 121(10):e2307876121. find
  42. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128:302–319. find
  43. Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 5184–5207. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oh-2025-impact-token-granularity-predictive-power/