Elamur Research⌕
☰
home / papers / 2024 / On the Proper Treatment of Tokenization in Psycholinguistics

On the Proper Treatment of Tokenization in Psycholinguistics

Summary

Psycholinguists often relate a language model's surprisal for a stretch of text to how hard readers find it, for example via gaze duration. Modern models, however, assign probabilities to token sequences produced by tokenizers such as BPE. The character spans that experimenters care about (regions of interest) frequently do not line up with those tokens. The authors argue that tokenization should be treated as irrelevant to the theory. Stimuli are character strings, so a token-level model should first be approximately marginalized into a character-level model. Its surprisal can then be read off for any character span, which the authors call a focal area. They apply the approximate marginalization of Vieira et al. (2024) to four English eye-tracking corpora and compare different focal areas as predictors. Surprisal of the whole region, with either leading or trailing whitespace, is seldom the strongest predictor. Partial or look-ahead spans often match or beat it. This opens a broader, tokenizer-independent space of predictors for reading research.

Key points

  • The paper challenges recent proposals to attach trailing whitespace to a word's surprisal. It argues these proposals reflect a BPE quirk, since whitespace appears only at the start of tokens. Eye-tracking practice usually attaches the preceding space instead.
  • On CELER, the surprisal of a region's first three characters predicts skip rate significantly better than full-region surprisal. Fixed-size focal areas with leading whitespace scored about 13% higher than those with trailing whitespace (p < 0.01).
  • On Provo and MECO, two kinds of focal area perform on par with the full region:
    • fixed-size initial spans;
    • dynamically sized spans, for example an 8-character rightward identification span.
  • On UCL, look-ahead focal areas that extend into the next region significantly improve predictions of first fixation, gaze and total duration. They also help for total duration on CELER.
  • The study has several limitations:
    • It covers only English L1 eye-tracking data and leaves out self-paced reading.
    • It uses linear models without mixed effects.
  • Code is available at github.com/rycolab/psycho-toke.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Psycholinguistics & language acquisition Tokenization & segmentation

Cite

Cite as Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira and Ryan Cotterell. 2024. On the Proper Treatment of Tokenization in Psycholinguistics. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.1032

@inproceedings{giulianelli2024proper,
  title = {{On the Proper Treatment of Tokenization in Psycholinguistics}},
  author = {Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira and Ryan Cotterell},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1032},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.1032}
}

References

86 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Doris Aaronson and Hollis Shapiro Scarborough. 1976. Performance theories for sentence coding: Some quantitative evidence. Journal of Experimental Psychology: Human Perception and Performance, 2(1). find
  2. Marco Baroni. 2022. On the proper role of linguistically oriented deep net analysis in linguistic theorising. In Algebraic Structures in Natural Language. CRC Press. find
  3. Lisa Beinborn and Yuval Pinter. 2023. Analyzing cognitive plausibility of subword tokenization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  4. Yevgeni Berzak, Chie Nakamura, Amelia Smith, Emily Weng, Boris Katz, Suzanne Flynn, and Roger Levy. 2022. CELER: A 365-Participant Corpus of Eye Movements in L1 and L2 English Reading. Open Mind. find
  5. Harry E. Blanchard, Alexander Pollatsek, and Keith Rayner. 1989. The acquisition of parafoveal word information in reading. Perception & Psychophysics, 46(1). find
  6. Veronica Boyce, Roger Levy, Veronica Boyce, and Roger P. Levy. 2023. A-maze of Natural Stories: Comprehension and surprisal in the Maze task. Glossa Psycholinguistics, 2(1). find
  7. Marc Brysbaert, Boris New, and Emmanuel Keuleers. 2012. Adding part-of-speech information to the SUBTLEX-US word frequencies. Behavior research methods, 44. find
  8. Kris Cao and Laura Rimell. 2021. You should evaluate your language model on marginal likelihood over tokenisations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  9. Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. 2023. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature Human Behaviour, 7(3). find
  10. Nadezhda Chirkova, Germán Kruszewski, Jos Rozen, and Marc Dymetman. 2023. Should you marginalize over possible tokenizations? In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  11. Andrea Gregor de Varda, Marco Marelli, and Simona Amenta. 2023. Cloze probability, predictability ratings, and computational estimates for 205 English sentences, aligned with existing EEG and reading time data. Behavior Research Methods. find
  12. Rudy den Buurman, Theo Roersema, and Jack F. Gerris-sen. 1981. Eye movements and the perceptual span in reading. Reading Research Quarterly, 16(2). find
  13. Emanuel Donchin. 1979. Event-related brain potentials: A tool in the study of human information processing. In Evoked Brain Potentials and Behavior. Springer. find
  14. Kelly Enochson and Jennifer Culbertson. 2015. Collecting psycholinguistic response time data using Amazon Mechanical Turk. PloS one, 10(3). find
  15. Dennis F. Fisher. 1975. Reading and visual search. Memory & Cognition, 3(2). find
  16. Kenneth I. Forster, Christine Guerrera, and Lisa Elliot. 2009. The maze task: Measuring forced incremental sentence processing time. Behavior Research Methods, 41. find
  17. Stefan L. Frank. 2024. Neural language model gradients predict event-related brain potentials. In Proceedings of the Society for Computation in Linguistics 2024. find
  18. Stefan L. Frank, Irene Fernandez Monsalve, Robin L. Thompson, and Gabriella Vigliocco. 2013. Reading time data for evaluating broad-coverage models of English sentence processing. Behavior Research Methods, 45. find
  19. Richard Futrell, Edward Gibson, and Roger P. Levy. 2020. Lossy-context surprisal: An information-theoretic model of memory effects in sentence processing. Cognitive Science, 44(3). find
  20. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics… our card
  21. Juan Luis Gastaldi, John Terilla, Luca Malagutti, Brian DuSell, Tim Vieira, and Ryan Cotterell. 2024. The foundations of tokenization: Statistical and computational concerns. Preprint, arXiv:2407.11606. — [arXiv:2407.11606](https://arxiv.org/abs/2407.11606) find
  22. Renato Lui Geh, Honghua Zhang, Kareem Ahmed, Benjie Wang, and Guy Van den Broeck. 2024. Where is the signal in tokenization space? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. find
  23. Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. 2004. Bayesian Data Analysis. Chapman and Hall/CRC. find
  24. Edward Gibson. 2001. The Dependency Locality Theory: A Distance-Based Theory of Linguistic Complexity. In Image, Language, Brain: Papers from the First Mind Articulation Project Symposium. The MIT Press. find
  25. Mario Giulianelli, Andreas Opedal, and Ryan Cotterell. 2024a. Generalized measures of anticipation and responsivity in online language processing. In Findings of the Association for Computational Linguistics: EMNLP 2024. find
  26. Mario Giulianelli, Arabella Sinclair, and Raquel Fernández. 2022. Construction repetition reduces information rate in dialogue. In Proceedings of the Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Pro… find
  27. Mario Giulianelli, Sarenne Wallbridge, Ryan Cotterell, and Raquel Fernández. 2024b. Incremental alternative sampling as a lens into the temporal and representational resolution of linguistic prediction. Preprint, PsyArXiv:10.31234. find
  28. Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics. our card
  29. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Meeting of the North American Chapter of the Association for Computational Linguistics. our card
  30. Martin Haspelmath. 2011. The indeterminacy of word segmentation and the nature of morphology and syntax. Folia Linguistica, 45(1). find
  31. Jacob Louis Hoover, Morgan Sonderegger, Steven T Piantadosi, and Timothy J O’Donnell. 2023. The plausibility of sampling as an algorithmic theory of sentence processing. Open Mind, 7. find
  32. J. Hyönä and R. K. Olson. 1995. Eye fixation patterns among dyslexic and normal readers: Effects of word length and word frequency. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(6). find
  33. Albert W. Inhoff and S. Tousman. 1990. Lexical integration across saccades in reading. Psychological research, 52(4). find
  34. Albrecht Werner Inhoff. 1989. Lexical access during eye fixations in reading: Are word access codes used to integrate lexical information across interword fixations? Journal of Memory and Language, 28(4). find
  35. Albrecht Werner Inhoff. 1990. Integrating information across eye fixations in reading: The role of letter and word units. Acta Psychologica, 73(3). find
  36. Albrecht Werner Inhoff and Keith Rayner. 1986. Parafoveal word processing during eye fixations in reading: Effects of word frequency. Perception & Psychophysics, 40(6). find
  37. Deborah N. Jakobi, Thomas Kern, David R. Reich, Patrick Haller, and Lena A. Jäger. 2024. PoTeC: A German naturalistic eye-tracking-while-reading corpus. CoRR, abs/2403.00506. find
  38. Marcel A. Just, Patricia A. Carpenter, and Jacqueline D. Woolley. 1982. Paradigms and processes in reading comprehension. Journal of Experimental Psychology: General, 111(2). find
  39. Alexandra Krauska and Ellen Lau. 2023. Moving away from lexicalism in psycho-and neuro-linguistics. Frontiers in Language Sciences, 2. find
  40. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3). find
  41. Susan D. Lima. 1987. Morphological analysis in sentence reading. Journal of Memory and Language, 26(1). find
  42. Susan D. Lima and Albrecht W. Inhoff. 1985. Lexical access during eye fixations in reading: effects of word-initial letter sequence. Journal of Experimental Psychology: Human Perception and Performance, 11(3). find
  43. Steven G. Luke and Kiel Christianson. 2018. The Provo Corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50(2). find
  44. Barbara C. Malt and John G. Seamon. 1978. Peripheral and cognitive components of eye guidance in filled-space reading. Perception & Psychophysics, 23(5). find
  45. Alec Marantz. 2001. Words. WCCFL XX Handout, USC. find
  46. G. W. McConkie, P. W. Kerr, M. D. Reddix, and D. Zola. 1988. Eye movement control during reading: I. the location of initial eye fixations on words. Vision Research, 28(10). find
  47. George W. McConkie and Keith Rayner. 1975. The span of the effective stimulus during a fixation in reading. Perception & Psychophysics, 17(6). find
  48. George W. McConkie and Keith Rayner. 1976. Asymmetry of the perceptual span in reading. Bulletin of the Psychonomic Society, 8(5). find
  49. George W. McConkie and David Zola. 1987. Visual attention during eye fixations while reading. In Attention and Performance XII: The psychology of reading. Routledge. find
  50. Clara Meister, Tiago Pimentel, Thomas Clark, Ryan Cotterell, and Roger Levy. 2022. Analyzing wrap-up effects through an information-theoretic lens. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  51. Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021. Revisiting the Uniform Information Density hypothesis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  52. Danny Merkx and Stefan L. Frank. 2021. Human sentence processing: Recurrence or attention? In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics. find
  53. James A. Michaelov, Megan D. Bardolph, Cyma K. Van Petten, Benjamin K. Bergen, and Seana Coulson. 2024. Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects. Neurobiology of Language, 5(1). find
  54. Sathvik Nair and Philip Resnik. 2023. Words, subwords, and morphemes: What really matters in the surprisal-reading time relationship? In Findings of the Association for Computational Linguistics: EMNLP. find
  55. Byung-Doh Oh, Christian Clark, and William Schuler. 2021. Surprisal estimators for human reading times need character models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing. find
  56. Byung-Doh Oh and William Schuler. 2023. Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? Transactions of the Association for Computational Linguistics, 11. our card
  57. Byung-Doh Oh and William Schuler. 2024. Leading whitespaces of language models’ subword vocabulary poses a confound for calculating word probabilities. Preprint, arXiv:2406.10851. — [arXiv:2406.10851](https://arxiv.org/abs/2406.10851) find
  58. J. Kevin O’Regan. 1980. The control of saccade size and fixation duration in reading: The limits of linguistic control. Perception & Psychophysics, 28. find
  59. Tiago Pimentel and Clara Meister. 2024. How to compute the probability of a word. Preprint, arXiv:2406.14561. find
  60. Alexander Pollatsek and Keith Rayner. 1982. Eye movement control in reading: The role of word boundaries. Journal of Experimental Psychology: Human Perception and Performance, 8(6). find
  61. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8). find
  62. Keith Rayner. 1975. The perceptual span and peripheral cues in reading. Cognitive Psychology, 7(1). find
  63. Keith Rayner. 1979. Eye guidance in reading: Fixation locations in words. Perception, 8. find
  64. Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. Psychological bulletin, 124(3). find
  65. Keith Rayner and James H. Bertera. 1979. Reading without a fovea. Science, 206(4417). find
  66. Keith Rayner, Kathryn H. Chace, Timothy J. Slattery, and Jane Ashby. 2006. Eye movements as reflections of comprehension processes in reading. Scientific studies of reading, 10(3). find
  67. Keith Rayner and Susan A. Duffy. 1986. Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & Cognition, 14(3). find
  68. Keith Rayner, Albrecht Werner Inhoff, Robert E. Morrison, Maria L. Slowiaczek, and James H. Bertera. 1981. Masking of foveal and parafoveal vision during eye fixations in reading. Journal of Experimental Psychology: Human Perception and Performance, 7(1). find
  69. Keith Rayner and Alexander Pollatsek. 1981. Eye movement control during reading: Evidence for direct control. The Quarterly Journal of Experimental Psychology Section A, 33(4). find
  70. Keith Rayner, Timothy J Slattery, Denis Drieghe, and Simon P Liversedge. 2011. Eye movements and word skipping during reading: Effects of word length and predictability. Journal of Experimental Psychology: Human Perception and Performance, 37(2). find
  71. Keith Rayner, Arnold D. Well, Alexander Pollatsek, and James H Bertera. 1982. The availability of useful information to the right of fixation in reading. Perception & Psychophysics, 31(6). find
  72. Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2021. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45)… find
  73. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. our card
  74. Cory Shain, Idan Asher Blank, Marten van Schijndel, William Schuler, and Evelina Fedorenko. 2020. fMRI reveals language-specific predictive coding during naturalistic sentence comprehension. Neuropsychologia, 138. find
  75. Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. Large-scale evidence for logarithmic effects of word predictability on reading time. Proceedings of the National Academy of Sciences, 121(10). find
  76. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3). find
  77. Antoine Tremblay, Bruce Derwing, Gary Libben, and Chris Westbury. 2011. Processing advantages of lexical bundles: Evidence from self-paced reading and sentence recall tasks. Language Learning, 61(2):569– 613. find
  78. N. R. Underwood and G. W. McConkie. 1985. Perceptual span for letter distinctions during reading. Reading Research Quarterly, 20(2). find
  79. Tim Vieira, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Mario Giulianelli, John Terilla, Timothy J. O’Donnell, and Ryan Cotterell. 2024. From language models over tokens to language models over characters. find
  80. Ethan G. Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023. Testing the Predictions of Surprisal Theory in 11 Languages. Transactions of the Association for Computational Linguistics, 11. our card
  81. Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. In Proceedings of the Annual Meeting of the Cognitive Science Society. Cognitive Science Society. find
  82. Simon N. Wood. 2004. Stable and efficient multiple smoothing parameter estimation for generalized additive models. Journal of the American Statistical Association, 99(467):673–686. find
  83. Simon N. Wood. 2017. Generalized Additive Models: An Introduction with R, 2nd edition. Chapman and Hall/CRC. find
  84. Jun Sen Yee, Mario Giulianelli, and Arabella J. Sinclair. 2024. Efficiency and effectiveness in task-oriented dialogue: On construction repetition, information rate, and task success. In Proceedings of the Joint International Conference on Computational Linguistics, Language Resources and Evaluation… find
  85. Noam Siegelman, Sascha Schroeder, Cengiz Acartürk, Hee-Don Ahn, Svetlana Alexeeva, Simona Amenta, Raymond Bertram, Rolando Bonandrini, Marc Brysbaert, Daria Chernova, et al. 2022. Expanding horizons of cross-linguistic research on reading: The multilingual eye-movement corpus (MECO). Behavior resear… find
  86. Nathaniel J. Smith and Roger Levy. 2008. Optimal processing times in reading: A formal model and empirical investigation. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 30. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/giulianelli-2024-proper-treatment-tokenization-psycholinguistics/