Elamur Research⌕
☰
home / papers / 2024 / How to Compute the Probability of a Word

How to Compute the Probability of a Word

Paper · 2024 · Tiago Pimentel and Clara Meister

Summary

Researchers who use language models to measure surprisal or perplexity usually care about words. Most modern LMs, however, predict subword tokens, and turning subword probabilities into word probabilities is harder than it looks. Pimentel and Meister formalise words, tokenisers and the distributions over them. They then derive correct formulas for a word's probability in context. With end-of-word-marking tokenisers, multiplying the probabilities of a word's subwords is enough. With beginning-of-word (bow) tokenisers, which GPT models, Pythia and Mistral use, the calculation must also account for whether the next token starts a new word. Many recent psycholinguistic studies skipped this step. The fix adds no cost, since it fits within one forward pass. The authors re-ran two earlier analyses with corrected probabilities: one on reading-based sentence comprehension and one on word lengths and communicative efficiency. The original conclusions held, but the measured quantities shifted in statistically significant ways. The method is therefore not a minor detail for empirical studies.

Key points

  • The common "bug" in bow-tokeniser LMs is corrected by renormalising with the probability mass of bow-marked tokens, both after the target word and after its context. The paper's Figure 1 gives this formula, and it comes as a pip package, wordsprobability (code: tpimentelms/probability-of-a-word).
  • Prior work hit by this bug includes studies by Wilcox et al. (2023a), Shain et al. (2024), Oh and Schuler (2023b) and Pimentel et al. (2023a). Concurrent work by Oh and Schuler (2024) found a similar fix independently.
  • On sentence comprehension, corrected surprisal predicted reading data only slightly better for individual models. Pooled across all models, the improvement was consistent and significant on all four datasets (α < 0.01, permutation tests). The same pattern held in seven other languages.
  • In the word-length study on English Wiki-40B, the correction lowered correlations for both channel capacity hypothesis predictors (Piantadosi et al.; Pimentel et al.). Zipf's frequency-based account remained the best predictor of word lengths.
  • Limitations include tokeniser corner cases the paper does not cover, such as Spanish "¿". The theory also assumes autoregressive models, and the empirical check covers only two studies.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Psycholinguistics & language acquisition · 102 works

Cite

Cite as Tiago Pimentel and Clara Meister. 2024. How to Compute the Probability of a Word. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.1020

@inproceedings{pimentel2024how,
  title = {{How to Compute the Probability of a Word}},
  author = {Tiago Pimentel and Clara Meister},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1020},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.1020}
}

References

49 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christian Bentz and Ramon Ferrer-i-Cancho. 2016. Zipf’s law of abbreviation as a language universal. In Proceedings of the Leiden Workshop on Capturing Phylogenetic Algorithms for Linguistics. Universität Tübingen. find
  2. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. Pythia: A suite for analyzing large language models across trainin… find
  3. Kris Cao and Laura Rimell. 2021. You should evaluate your language model on marginal likelihood over tokenisations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2104–2114, Online and Punta Cana, Dominican Republic. Association for Computational Lin… find
  4. Thomas M. Cover and Joy A. Thomas. 2006. Elements of Information Theory, second edition. Wiley-Interscience. find
  5. Robert M. W. Dixon and Alexandra Y. Aikhenvald. 2003. Word: a typological framework. In Word: A Cross-linguistic Typology, page 1–41. Cambridge University Press. find
  6. Li Du, Lucas Torroba Hennigen, Tiago Pimentel, Clara Meister, Jason Eisner, and Ryan Cotterell. 2023. A measure-theoretic characterization of tight language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9744–9770, To… find
  7. Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko. 2018. The natural stories corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation. European Language Resources Association. find
  8. Matthias Gallé. 2019. Investigating the effectiveness of BPE: The power of shorter sequences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1375–1381, Hong Kong… find
  9. Edward Gibson, Richard Futrell, Steven T. Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. 2019. How efficiency shapes human language. Trends in Cognitive Sciences, 23(5):389– 407. find
  10. Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira, and Ryan Cotterell. 2024. On the proper treatment of tokenization in psycholinguistics. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA. Association for Com…
  11. Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah. Association for Comput…
  12. Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. 2020. Wiki-40B: Multilingual language model dataset. In Proceedings of the 12th Language Resources and Evaluation Conference, page 2440–2452, Marseille, France. European Language Resources Association. find
  13. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics, pages 1–8.
  14. Tracy Alan Hall and Ursula Kleinhenz. 1999. Studies on the phonological word. John Benjamins. find
  15. Martin Haspelmath. 2023. Defining the word. WORD, 69(3):283–297. find
  16. Davis Howes. 1968. Zipf’s law and Miller’s random-monkey model. The American Journal of Psychology, 81(2):269–272. find
  17. Marcel Adam Just, Patricia A. Carpenter, and Jacqueline D. Woolley. 1982. Paradigms and processes in reading comprehension. Journal of Experimental Psychology: General, 111(2):228–238. find
  18. Alan Kennedy, Robin Hill, and Joel Pynte. 2003. The Dundee corpus. In Proceedings of the 12th European Conference on Eye Movements. find
  19. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ…
  20. Natalia Levshina. 2022. Frequency, informativity and word length: Insights from typologically diverse corpora. Entropy, 24(2). find
  21. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  22. Marina Nespor and Irene Vogel. 2007. Prosodic Phonology: With a New Foreword. De Gruyter Mouton, Berlin, Boston. find
  23. Byung-Doh Oh, Christian Clark, and William Schuler. 2022. Comparison of structural parsers and neural language models as surprisal estimators. Frontiers in Artificial Intelligence, 5. find
  24. Byung-Doh Oh and William Schuler. 2023a. Transformer-based language model surprisal predicts human reading times best with about two billion training tokens. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1915–1921, Singapore. Association for Computational Linguistic… find
  25. Byung-Doh Oh and William Schuler. 2023b. Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:336–350.
  26. Byung-Doh Oh and William Schuler. 2024. Leading whitespaces of language models’ subword vocabulary poses a confound for calculating word probabilities. arXiv preprint arXiv:2406.10851. — [arXiv:2406.10851](https://arxiv.org/abs/2406.10851) find
  27. Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2011. Word lengths are optimized for efficient communication. Proceedings of the National Academy of Sciences, 108(9):3526–3529. find
  28. Tiago Pimentel, Clara Meister, Ethan Wilcox, Kyle Mahowald, and Ryan Cotterell. 2023a. Revisiting the optimality of word lengths. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2240–2255, Singapore. Association for Computational Linguistics. find
  29. Tiago Pimentel, Clara Meister, Ethan G. Wilcox, Roger P. Levy, and Ryan Cotterell. 2023b. On the effect of anticipation on reading times. Transactions of the Association for Computational Linguistics, 11:1624–1642. find
  30. Roger Levy and T. Florian Jaeger. 2007. Speakers optimize information density through syntactic reduction. In Advances in Neural Information Processing Systems, volume 19. MIT Press. find
  31. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
  32. Steven G. Luke and Kiel Christianson. 2018. The Provo corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50(2):826– 833. find
  33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
  34. Stephan C. Meylan and Thomas L. Griffiths. 2021. The challenges of large-scale, web-based language datasets: Word length and predictability revisited. Cognitive Science, 45(6):e12983. find
  35. Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Samson Tan. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP. arXiv preprint arXiv:2112.10508.
  36. Cory Shain. 2019. A large-scale study of the effects of word frequency and predictability in naturalistic reading. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), page… find
  37. Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. Large-scale evidence for logarithmic effects of word predictability on reading time. Proceedings of the National Academy of Sciences, 121(10):e2307876121. find
  38. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022. mGPT: Few-shot learners go multilingual. arXiv preprint arXiv:2204.07580.
  39. Noam Siegelman, Sascha Schroeder, Cengiz Acartürk, Hee-Don Ahn, Svetlana Alexeeva, Simona Amenta, Raymond Bertram, Rolando Bonandrini, Marc Brysbaert, Daria Chernova, Sara Maria Da Fonseca, Nicolas Dirix, Wouter Duyck, Argyro Fella, Ram Frost, Carolina A. Gattei, Areti Kalaitzi, Nayoung Kwon, Kaidi… find
  40. Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Me…
  41. George K. Zipf. 1935. The Psychobiology of Language. London: Routledge. find
  42. Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207, Toronto, Canada. Association for… find
  43. Nathaniel J. Smith and Roger Levy. 2008. Optimal processing times in reading: a formal model and empirical investigation. In Proceedings of the Cognitive Science Society, volume 30, pages 595–600. find
  44. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319. find
  45. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. arXiv pr…
  46. Ethan Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. In Proceedings of the Cognitive Science Society. find
  47. Ethan Wilcox, Clara Meister, Ryan Cotterell, and Tiago Pimentel. 2023a. Language model quality correlates with psychometric predictive power in multiple languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7503–7511, Singapore. Association for C… find
  48. Ethan Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023b. Testing the predictions of surprisal theory in 11 languages. Transactions of the Association for Computational Linguistics, 11:1451–1470.
  49. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pimentel-2024-how-compute-probability-word/