Elamur Research⌕
☰
home / papers / 2026 / Reading Time in the Wild: An Assessment of Readability Predictors Based on Naturally-Observed Reading Times

Reading Time in the Wild: An Assessment of Readability Predictors Based on Naturally-Observed Reading Times

Summary

Readability research often uses reading time as a proxy for text difficulty, but most of that evidence comes from lab studies with eye-tracking or self-paced reading. This paper asks whether those findings hold for unsupervised, everyday online reading. Through a partnership with the Mediahuis publishing group, the authors obtained clickstream data for Dutch news articles published in 2024. They converted aggregated reading time into average seconds per token and removed duplicates, skimmed articles and paywalled items, which left 2,929 regional articles. Candidate predictors were grouped into four dimensions: text profiling (Profiling-UD, T-Scan), readability metrics (LiNT, Flesch-Douma, Brouwer Index), LLM surprisal, and article metadata. Each was tested for correlation with per-token reading time and for predictive power. None of the proxies correlated meaningfully with reading time, and none supported reliable prediction. The authors also used an LLM to simplify the texts to CEFR B1. Unexpectedly, the models then predicted longer per-token reading times. The authors conclude that clickstream reading time should be treated cautiously as a signal of readability or comprehension.

Key points

  • Data and filtering: The raw data covered 24,946 Mediahuis articles from 2024. Articles were kept only if they had at least 100 words, at least 25 views and a reading time of at least 0.15 s per token. This left 2,929 articles from Mediahuis Noord.
  • Feature dimensions:
    • Text profiling: Profiling-UD and T-Scan.
    • Readability metrics: LiNT, Flesch-Douma and Brouwer Index.
    • LLM surprisal: tweety-7b-dutch-v24a, Qwen3-4B, fietje-2 and mGPT.
    • Metadata: topic, newspaper and day of week.
  • Contrast with existing corpora: Unlike the controlled eye-tracking corpora RaCCooNS, MECO and GECO, this dataset reflects uncontrolled reading behaviour averaged over thousands of readers.
  • Simplification effects: B1-level LLM simplification lowered many syntactic measures, such as dependency lengths and relative clauses. However, it raised surprisal scores in most texts (mean relative change of about +0.5 to +0.6), and predicted reading time per token went up.
  • Availability: Code is published on GitHub (sijbrenvv/Reading-time-in-the-wild). The dataset is available only on request, under a license and a non-disclosure agreement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/56xa82ywv9us

@inproceedings{vaals2026reading,
  title = {{Reading Time in the Wild: An Assessment of Readability Predictors Based on Naturally-Observed Reading Times}},
  author = {Vaals, Sijbren van and Noord, Rik van and Nissim, Malvina},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/56xa82ywv9us},
  url = {https://doi.org/10.63317/56xa82ywv9us}
}

References

52 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nadine Beks van Raaij, Daan Kolkman, and Ksenia Podoynitsyna. 2024. Clearer governmental communication: Text simplification with Chat-GPT evaluated by quantitative and qualitative research. In Proceedings of the Workshop on DeTermIt! Evaluating Text Difficulty in a Multilingual Context @ LREC-COLING… find
  2. Luca Benedetto, Gabrielle Gaudeau, Andrew Caines, and Paula Buttery. 2025. Assessing how accurately large language models encode and apply the common european framework of reference for languages. Computers and Education: Artificial Intelligence, 8:100353. find
  3. Wilbert Berendsen. 2013. Custom Python library to hyphenate text using existing Hunspell hyphenation dictionaries. find
  4. Sam Boeve and Louisa Bogaerts. 2025. A systematic evaluation of Dutch large language models’ surprisal estimates in sentence, paragraph, and book reading. Behavior Research Methods, 57(266). find
  5. R.H.M. Brouwer. 1963. Onderzoek naar de leesmoeilijkheden van Nederlands proza. Pedagogische Studiën, 40. find
  6. Dominique Brunato, Andrea Cimino, Felice Dell’Orletta, Giulia Venturi, and Simonetta Montemagni. 2020. Profiling-UD: a tool for linguistic profiling of texts. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7145–7151, Marseille, France. European Language Resources A… find
  7. Dominique Brunato, Lorenzo De Mattei, Felice Dell’Orletta, Benedetta Iavarone, and Giulia Venturi. 2018. Is this sentence difficult? Do you agree? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2690–2699, Brussels, Belgium. Association for Computatio… find
  8. Bram Bulté, Leen Sevens, and Vincent Vandeghinste. 2018. Automating lexical simplification in Dutch. Computational Linguistics in the Netherlands Journal, 8:24–48. find
  9. Uschi Cop, Nicolas Dirix, Denis Drieghe, and Wouter Duyck. 2017. Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading. Behavior research methods, 49(2):602–615. find
  10. Orphée De Clercq and Véronique Hoste. 2016. All mixed up? Finding the optimal feature set for general readability prediction and its application to English and Dutch. Computational Linguistics, 42(3):457–490. find
  11. Tovly Deutsch, Masoud Jasbi, and Stuart Shieber. 2020. Linguistic features for readability assessment. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–17, Seattle, WA, USA → Online. Association for Computational Linguistics. find
  12. Ir WH Douma. 1960. De leesbaarheid van land-bouwbladen. Afdelingen voor Sociale Wetenschappen aan de Landbouwhogeschool. find
  13. Rudolph Flesch. 1948. A new readability yardstick. Journal of applied psychology, 32(3):221. find
  14. Stefan L Frank and Anna Aumeistere. 2024. An eye-tracking-with-EEG coregistration corpus of narrative sentences. Language Resources and Evaluation, 58(2):641–657. find
  15. Edward Gibson. 1998. Linguistic complexity: locality of syntactic dependencies. Cognition, 68:1– 76. find
  16. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics. find our card
  17. Karin Harbusch and Ina Steinmetz. 2022. A computer-assisted writing tool for an extended variety of leichte sprache (easy-to-read German). Frontiers in Communication, Volume 6 - 2021. find
  18. Eliza Hobo, Charlotte Pouw, and Lisa Beinborn. 2023. “Geen makkie”: Interpretable classification and simplification of Dutch text complexity. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 503–517, Toronto, Canada. Association for… find
  19. Nora Hollenstein, Itziar Gonzalez-Dios, Lisa Beinborn, and Lena Jäger. 2022. Patterns of text readability in human and predicted eye movements. In Proceedings of the Workshop on Cognitive Aspects of the Lexicon, pages 1–15, Taipei, Taiwan. Association for Computational Linguistics. find
  20. Kuan-Jung Huang, Suhas Arehalli, Mari Kugemoto, Christian Muxica, Grusha Prasad, Brian Dillon, and Tal Linzen. 2024. Large-scale benchmark yields no evidence that language model surprisal explains syntactic disambiguation difficulty. Journal of Memory and Language, 137:104510. find
  21. Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ricardo Baeza-Yates. 2017. Story-focused reading in online news and its potential for user engagement. Journal of the Association for Information Science and Technology, 68(4):869–883. find
  22. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  23. Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 4768–4777, Red Hook, NY, USA. Curran Associates Inc. find our card
  24. Henk Pander Maat, Suzanne Kleijn, and Servaas Frissen. 2023. LiNT: een leesbaarheidsformule en een leesbaarheidsinstrument. Tijdschrift voor Taalbeheersing, 45(1):2–39. find
  25. Louis Martin, Éric de la Clergerie, Benoît Sagot, and Antoine Bordes. 2020. Controllable sentence simplification. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4689–4698, Marseille, France. European Language Resources Association. find
  26. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. EuroLLM: Multilingual language m… find our card
  27. Kanishka Misra. 2022. Minicons: Enabling flexible behavioral and representational analyses of transformer language models. arXiv preprint arXiv:2203.13112. find
  28. Amy Mitchell, Galen Stocking, and Katerina Eva Matsa. 2016. Long-form reading shows signs of life in our mobile news world. Technical report, Pew Research Center. Report by the Pew Research Center in association with the John S. and James L. Knight Foundation. find
  29. Joakim Nivre. 2015. Towards a universal grammar for natural language processing. In Computational Linguistics and Intelligent Text Processing, pages 3–16, Cham. Springer International Publishing. find
  30. Byung-Doh Oh and Tal Linzen. 2025. To model human linguistic prediction, make LLMs less superhuman. find
  31. Nelleke Oostdijk, Martin Reynaert, Véronique Hoste, and Ineke Schuurman. 2013. The Construction of a 500-Million-Word Reference Corpus of Contemporary Written Dutch, pages 219– 247. Springer Berlin Heidelberg, Berlin, Heidelberg. find
  32. H.L.W. Pander Maat and N. Dekker. 2016. Tek-stgenres analyseren op lexicale complexiteit met TScan. Tijdschrift voor taalbeheersing, 38(3):263–304. find
  33. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
  34. Qwen Team. 2025. Qwen3 technical report. find
  35. Sidharth Ranjan, Marten van Schijndel, Sumeet Agarwal, and Rajakrishnan Rajkumar. 2022. Dual mechanism priming effects in Hindi word order. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on… find
  36. François Remy, Pieter Delobelle, Hayastan Avetisyan, Alfiya Khabibullina, Miryam de Lhoneux, and Thomas Demeester. 2024. Trans-tokenization and cross-lingual vocabulary transfers: Language adaptation of LLMs for low-resource NLP. find
  37. Andreas Säuberli, Franz Holzknecht, Patrick Haller, Silvana Deilen, Laura Schiffl, Silvia Hansen-Schirra, and Sarah Ebling. 2024. Digital comprehensibility assessment of simplified texts among persons with intellectual disabilities. In Proceedings of the 2024 CHI Conference on Human Factors in Compu… find
  38. Sarah Schwarm and Mari Ostendorf. 2005. Reading level assessment using support vector machines and statistical language models. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 523–530, Ann Arbor, Michigan. Association for Computational Lingu… find our card
  39. Theresa Seidl and Vincent Vandeghinste. 2024. Controllable sentence simplification in Dutch. Computational Linguistics in the Netherlands Journal, 13:31–61. find
  40. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2023. mGPT: Few-shot learners go multilingual. find our card
  41. Noam Siegelman, Sascha Schroeder, Cengiz Acartürk, Hee-Don Ahn, Svetlana Alexeeva, Simona Amenta, Raymond Bertram, Rolando Bonandrini, Marc Brysbaert, Daria Chernova, et al. 2022. Expanding horizons of cross-linguistic research on reading: The multilingual eye-movement corpus (MECO). Behavior resear… find
  42. Abhinav Deep Singh, Poojan Mehta, Samar Husain, and Rajkumar Rajakrishnan. 2016. Quantifying sentence complexity based on eye-tracking measures. In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC), pages 202–212, Osaka, Japan. find
  43. The COLING 2016 Organizing Committee. find
  44. Nathaniel J Smith and Roger Levy. 2008. Optimal processing times in reading: A formal model and empirical investigation. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 30. find
  45. C. Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72–101. find
  46. Sowmya Vajjala and Detmar Meurers. 2012. On improving the accuracy of readability classification using insights from second language acquisition. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 163–173, Montréal, Canada. Association for Computational Ling… find our card
  47. Philip van Oosten, Dries Tanghe, and Véronique Hoste. 2010. Towards an improved methodology for automated readability prediction. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). find
  48. Bram Vanroy. 2024. Fietje: An open, efficient LLM for Dutch. find
  49. Daphne Wang, Mehrnoosh Sadrzadeh, Miloš Stanojević, Wing-Yee Chow, and Richard Breheny. 2024. How can large language models become more human? In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 166–176, Bangkok, Thailand. Association for Computational Linguisti… find
  50. Ethan G. Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023. Testing the predictions of surprisal theory in 11 languages. Transactions of the Association for Computational Linguistics, 11:1451–1470. find our card
  51. Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80– 83. find
  52. Weijie Xu, Jason Chon, Tianran Liu, and Richard Futrell. 2023. The linearity of the effect of surprisal on reading times across languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15711–15721, Singapore. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vaals-2026-reading-time-wild-assessment-readability/