Elamur Research⌕
☰
home / papers / 2026 / Word predictability estimates from language models are not robust to tokenizer vocabulary

Word predictability estimates from language models are not robust to tokenizer vocabulary

Summary

Psycholinguists often take word surprisal from pretrained language models to predict human reading behavior. Those models, however, contain engineering choices that were never meant for cognitive modeling. This paper looks at one of them: the vocabulary size of a BPE tokenizer. The authors trained BPE tokenizers of many sizes on the BabyLM Corpus and applied them to the Natural Stories Corpus. They then checked how often words get split and whether the splits are morphologically sensible.

Next, they trained 5-gram, LSTM, and GPT-2 models with several vocabulary sizes and compared each word's surprisal. In principle, a word's surprisal should not depend on how it is segmented. Finally, they used LSTM surprisals to predict reading times.

Smaller vocabularies produce many more splits, but the share of morphologically valid splits stays roughly constant. Surprisal estimates nonetheless shift with vocabulary size, especially for words that are split. Reading times for split words are systematically mispredicted relative to unsplit words. The practical lesson is that experimental stimuli should take tokenization into account, for example by avoiding words the tokenizer splits.

Key points

  • Experiment 1 used 100 BPE tokenizers trained on the 100M-word BabyLM Corpus. Vocabulary sizes ran from 257 to 50257, the GPT-2 size, in steps of 500.
  • Words in the Natural Stories Corpus were labeled Vocab, Morphological, Alien, or NA using the umLabeller (Batsuren et al., 2024).
  • The tokens-per-word ratio drops steeply as vocabulary grows and levels off at around 5000. Most splits at any size are Alien (non-morphological). At 50257, BPE behaves essentially like a word-level tokenizer.
  • Surprisals from the 5-gram, LSTM, and GPT-2 models were not robust to vocabulary size. The largest effects fell on words whose split status changes between vocabulary sizes. The LSTM findings were also tested on the monolingual part of GECO, an eye-tracking corpus.
  • Surprisal-based reading-time models consistently under- or overpredict reading times for split words. The study is limited to English and BPE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.conll-main.3

@inproceedings{nguyen2026word,
  title = {{Word predictability estimates from language models are not robust to tokenizer vocabulary}},
  author = {Nguyen, Kien and Arehalli, Suhas},
  booktitle = {Proceedings of the 30th Conference on Computational Natural Language Learning},
  year = {2026},
  doi = {10.18653/v1/2026.conll-main.3},
  url = {https://doi.org/10.18653/v1/2026.conll-main.3}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Suhas Arehalli and Tal Linzen. 2024. Neural Networks as Cognitive Models of the Processing of Syntactic Constraints. Open Mind, 8:558–614. find
  2. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui… find our card
  3. Khuyagbaatar Batsuren, Ekaterina Vylomova, Verna Dankers, Tsetsuukhei Delgerbaatar, Omri Uzan, Yuval Pinter, and Gábor Bella. 2024. Evaluating subword tokenization: Alien subword composition and OOV generalization challenge. Preprint, arXiv:2404.13292. — [arXiv:2404.13292](https://arxiv.org/abs/2404.13292) find
  4. Lisa Beinborn and Yuval Pinter. 2023. Analyzing cognitive plausibility of subword tokenization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4478–4486, Singapore. Association for Computational Linguistics. find
  5. Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira, and Ryan Cotterell. 2024. On the proper treatment of tokenization in psycholinguistics. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18556–18572, Miami, Florida, USA.… find
  6. Bastian Bunzeck and Sina Zarrieß. 2025. Subword models struggle with word learning, but surprisal hides Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. Preprint, arXiv:1803.11138. find our card
  7. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3464–3472, Miami, Florida, USA. Association for Computational Linguistics. find
  8. Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland. Association for Computational Linguistics. find our card
  9. Byung-Doh Oh and William Schuler. 2025. The impact of token granularity on the predictive power of language model surprisal. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4150–4162, Vienna, Austria. Association for Computati… find
  10. Kuan-Jung Huang, Suhas Arehalli, Mari Kugemoto, Christian Muxica, Grusha Prasad, Brian Dillon, and Tal Linzen. 2024. Large-scale benchmark yields no evidence that language model surprisal explains syntactic disambiguation difficulty. Journal of Memory and Language, 137:104510. find
  11. Tiago Pimentel and Clara Meister. 2024. How to compute the probability of a word. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18358–18375, Miami, Florida, USA. Association for Computational Linguistics. find
  12. Tatsuki Kuribayashi, Yohei Oseki, Souhaib Ben Taieb, Kentaro Inui, and Timothy Baldwin. 2025. Large Language Models Are Human-Like Internally. Transactions of the Association for Computational Linguistics, 13:1743–1766. find
  13. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  14. Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… find our card
  15. Jenny R Saffran. 2001. Words in a sea of sounds: The output of infant statistical learning. Cognition, 81(2):149–169. find
  16. Jenny R. Saffran, Richard N. Aslin, and Elissa L. Newport. 1996. Statistical learning by 8-month-old infants. Science, 274(5294):1926–1928. find
  17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin… find
  18. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  19. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. find our card
  20. Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. Large-scale evidence for logarithmic effects of word predictability on reading time. Proceedings of the National Academy of Sciences, 121(10):e2307876121. find
  21. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. find
  22. Nathaniel J Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3). find
  23. Sathvik Nair and Philip Resnik. 2023. Words, subwords, and morphemes: What really matters in the surprisal-reading time relationship? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11251–11260, Singapore. Association for Computational Linguistics. find
  24. Marten van Schijndel and Tal Linzen. 2021. Single-stage prediction models do not explain the magnitude of syntactic disambiguation difficulty. Cognitive Science, 45(6):e12988. find
  25. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the… find
  26. Byung-Doh Oh and William Schuler. 2023. Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:336–350. find our card
  27. Byung-Doh Oh and William Schuler. 2024. Leading whitespaces of language models’ subword vocabulary pose a confound for calculating word probabilities. find
  28. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler–gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221, Brussels, Belgium. Association for Computati… find our card
  29. Ethan G. Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023. Testing the Predictions of Surprisal Theory in 11 Languages. Transactions of the Association for Computational Linguistics, 11:1451–1470. find our card
  30. Ethan Gotlieb Wilcox, Richard Futrell, and Roger Levy. 2024. Using Computational Models to Test Syntactic Learnability. Linguistic Inquiry, 55(4):805–848. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2026-word-predictability-estimates-language-models/