Elamur Research⌕
☰
home / papers / 2026 / Getting Close to Cloze: Investigating Language Model and Human Cloze-test Performance in Afrikaans

Getting Close to Cloze: Investigating Language Model and Human Cloze-test Performance in Afrikaans

Paper · 2026 · Susan Lotz, Rik van Noord and Gertjan van Noord

Summary

Many languages, including Afrikaans, have weak or no tools for automatic readability assessment, and building such tools usually requires costly human cloze-test data. The authors test whether encoder-only language models could stand in for human participants. They use an Afrikaans cloze-test set that was designed for readability research and already answered by people in earlier work (Lotz et al., forthcoming). Several multilingual and Afrikaans-specific masked LMs fill in the same blanks, with several gaps per sentence. The main criterion is how closely each model's behaviour tracks the human answers, not its raw accuracy.

Model outputs correlate moderately with human responses. The strongest agreement comes from afRoBERTa, a relatively small monolingual model, rather than from the largest or most accurate system. A part-of-speech breakdown shows that the agreement is spread across word classes. Manual annotation shows that when both humans and the model miss, the model's guesses are usually less fitting.

The authors present this as the first human–LM cloze correlation study outside English. It supports using LMs for readability modelling in low-resource languages.

Key points

  • The best-correlating model is afRoBERTa, a RoBERTa-based LM trained only on Afrikaans (Eiselen, 2023, available via SADiLaR), with Pearson r = 0.62 and Spearman ρ = 0.62.
  • A fine-grained XPOS analysis using the Afrikaans tagset finds per-tag correlations mostly between about 0.2 and 0.6, so no small group of word classes drives the overall result. One tag even shows a negative correlation.
  • Four annotators judged wrong answers on a five-level scale: match/spelling variant, synonym, plausible fit, alternative fit, and incorrect. One annotator pair reached a weighted Cohen's κ of 0.64.
  • The human cloze data come from an earlier attempt to build a traditional readability formula for Afrikaans, which did not produce a reliable formula.
  • Afrikaans has over 6.5 million speakers and is classed as a "rising star" (level 3 of 0–6) in the Joshi et al. (2020) language-resource taxonomy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GroningenGroningen · Netherlands70 works · 5,142 citations🎓Stellenbosch UniversityStellenbosch · South Africa17 works · 24,141 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Susan Lotz, Rik van Noord and Gertjan van Noord. 2026. Getting Close to Cloze: Investigating Language Model and Human Cloze-test Performance in Afrikaans. Proceedings of Resources for African Indigenous Languages (RAIL) 2026 @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4th5sqwn9wjb

@inproceedings{lotz2026getting,
  title = {{Getting Close to Cloze: Investigating Language Model and Human Cloze-test Performance in Afrikaans}},
  author = {Susan Lotz, Rik {van Noord} and Gertjan {van Noord}},
  booktitle = {Proceedings of Resources for African Indigenous Languages (RAIL) 2026 @ LREC 2026},
  year = {2026},
  doi = {10.63317/4th5sqwn9wjb},
  url = {https://doi.org/10.63317/4th5sqwn9wjb}
}

References

67 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Alcoba Inciarte. 2023. SERENGETI: Massively multilingual language models for Africa. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1498–1537, Toronto, Canada. Association for Computational Linguistics. find
  2. Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336–4349, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. ↗
  3. Liesbeth Augustinus, Peter Dirix, Daniel van Niekerk, Ineke Schuurman, Vincent Vandeghinste, Frank Van Eynde, and Gerhard van Huyssteen. 2016. AfriBooms: An online treebank for Afrikaans. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 677–682, Portorož, Slovenia. European Language Resources Association (ELRA). find
  4. Lyle F Bachman. 1985. Performance on cloze tests with fixed-ratio and rational deletions. TESOL Quarterly, 19(3):535–556. ↗
  5. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The Belebele Benchmark: A parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand. Association for Computational Linguistics.
  6. Marc Benzahra and François Yvon. 2019. Measuring text readability with machine comprehension: a pilot study. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 412–422, Florence, Italy. Association for Computational Linguistics. find
  7. John R. Bormuth. 1967. The cloze readability procedure: A review of research on its use for evaluating instructional materials. Research Report ED010983, University of California, Los Angeles, CRESST / Centre for the Study of Evaluation. find
  8. John R. Bormuth. 1968. Cloze as a measure of readability: Criterion-reference scores. Yearbook of the International Reading Association, 17:303– 317. find
  9. Patricia L Carrell, Joan J. Carson, and Dong Zhe. 1993. First and second language reading strategies: Evidence from cloze. Reading in a Foreign Language, 10(1):953–65. ↗
  10. Kevyn Collins-Thompson. 2014. Computational assessment of text readability. ITL-International Journal of Applied Linguistics, 165(2):97–135. ↗
  11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics. ↗
  12. Jac Conradie and Anna Coetzee. 2014. 47. Afrikaans, pages 897–918. De Gruyter Mouton, Berlin, Boston. find
  13. Scott A Crossley, Stephen Skalicky, and Mihai Dascalu. 2019. Moving beyond classic readability formulas: New methods and new models. Journal of Research in Reading, 42(3-4):541–561. ↗
  14. Achmat Davids. 1994. Afrikaans—die produk van akkulturasie. In G. Olivier and A. Coetzee, editors, Nuwe perspektiewe op die geskiedenis van Afrikaans, pages 110–119. Southern Boekuitgewers, Pretoria. find
  15. Wietse De Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTje: A Dutch BERT model. arXiv preprint arXiv:1912.09582. ↗
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  17. Peter Dirix. 2023. The need for a large(r) Afrikaans treebank. Stellenbosch Papers in Linguistics Plus, 67. ↗
  18. Bonaventure F. P. Dossou, Atnafu Lambebo Tonja, Oreen Yousuf, Salomey Osei, Abigail Oppong, Iyanuoluwa Shode, Oluwabusayo Olufunke Awoyomi, and Chris Emezue. 2022. AfroLM: A self-active learning-based multilingual pretrained language model for 23 African languages. In Proceedings of the Third Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pages 52–64, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. ↗
  19. William H. DuBay. 2004. The principles of readability. Report, Impact Information, Costa Mesa, CA. Available at ERIC. Accessed 20 September 2022. find
  20. Roald Eiselen. 2023. NCHLT Afrikaans RoBERTa language model. SADiLaR Language Resource Repository, License: Creative Commons Attribution 4.0 International (CC-BY 4.0). Accessed 5 Oct 2023. find
  21. Roald Eiselen and Tanja Gaustad. 2023. Deep learning and low-resource languages: How much data is enough? A case study of three linguistically distinct South African languages. In Proceedings of the Fourth Workshop on Resources for African Indigenous Languages (RAIL 2023), pages 42–53. Association for Computational Linguistics. find
  22. Thomas François. 2015. When readability meets computational linguistics: A new paradigm in readability. Revue française de linguistique appliquée, (2):79–97. ↗
  23. Anna S. Gellert and Carsten Elbro. 2013. Activation of background knowledge for inference making: Effects on reading comprehension. Scientific Studies of Reading, 17(5):435–452. ↗
  24. A. Goldstein, Z. Zada, E. Buchnik, M. Schain, A. Price, B. Aubrey, S. A. Nastase, A. Feder, D. Emanuel, A. Cohen, A. Jansen, H. Gazula, G. Choe, A. Rao, C. Kim, C. Casto, L. Fanda, W. Doyle, D. Friedman, P. Dugan, L. Melloni, R. Reichart, S. Devore, A. Flinker, L. Hasenfratz, O. Levy, A. Hassidim, M. P. Brenner, Y. Matias, K. A. Norman, O. Devinsky, and U. Hasson. 2022. Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25:369–380. ↗
  25. K. S. Goodman. 1967. Reading: A psycholinguistic guessing game. Journal of the Reading Specialist, 6:126–135. ↗
  26. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. CoRR, abs/2111.09543. ↗
  27. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28, pages 1693–1701. ↗
  28. Markus J Hofmann, Steffen Remus, Chris Biemann, Ralph Radach, and Lars Kuchinke. 2022. Language models explain word reading times better than empirical predictability. Frontiers in Artificial Intelligence, 4:730570. ↗
  29. Raymond Joseph Horton. 1974. The construct validity of cloze procedure: An exploratory factor analysis of cloze, paragraph reading, and structure-of-intellect tests. Reading Research Quarterly, 10(2):248–251. find
  30. Cassandra L Jacobs, Loïc Grobol, and Alvin Tsang. 2024. Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned. arXiv preprint arXiv:2410.12057. ↗
  31. Carel Jansen, Rose Richards, and Liezl Van Zyl. 2017. Evaluating four readability formulas for Afrikaans. Stellenbosch Papers in Linguistics Plus, 53:149–166. ↗
  32. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020a. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64– 77. ↗
  33. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020b. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.
  34. Tom Kalinsky, Assaf Kasirer, and Yoav Goldberg. 2023. Simple and effective multi-token completion from masked language models. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2345–2359. Association for Computational Linguistics. find
  35. Jan-Christoph Kalo and Leandra Fichtel. 2022. Kamel: Knowledge analysis with multitoken entities in language models. In Automated Knowledge Base Construction (AKBC). find
  36. Judith M. H. Kamalski. 2007. Coherence Marking, Comprehension and Persuasion: On the Processing and Representation of Discourse. Ph.D. thesis, Utrecht University. find
  37. Suzanne Kleijn. 2018. Clozing in on readability: How linguistic features affect and predict text comprehension and on-line processing. Ph.D. thesis, Utrecht University. find
  38. Miyoko Kobayashi. 2002. Cloze tests revisited: Exploring item characteristics with special attention to scoring methods. The Modern Language Journal, 86(4):571–586. ↗
  39. Ernst Kotzé. 2018. Die klassifikasie van Afrikaans. LitNet (Menings). Published 18 April 2018. Accessed: 2026-02-16. find
  40. Anna Laurinavichyute, Irina A Sekerina, Kristina Bagdasaryan, Svetlana Alexeeva, and Nikita Zmanovksy. 2017. Russian sentence corpus: Benchmark measures of eye movements in reading in cyrillic. International Journal of Corpus Linguistics. find
  41. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  42. Adrielli Tina Lopes Rego, Joshua Snell, and Martijn Meeter. 2024. Language models outperform cloze predictability in a cognitive model of reading. PLOS Computational Biology, 20(9):e1012117. Susan Lotz, Bo Blankers, Rik van Noord, and Carel Jansen. forthcoming. Readability assessment in Afrikaans: Cloze scores, linguistic features and cross-validated linear regression. ↗
  43. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmBERT: A modern multilingual encoder with annealed language learning. arXiv preprint arXiv:2509.06888. ↗
  44. Cindy McKellar. 2022. Autshumato English-Afrikaans parallel corpora. SADiLaR Language Resource Repository. find
  45. Tebatso Gorgina Moape, Fulufhelo Mthombeni, and Annemie Stoman. 2025. Evaluating the impact of data scarcity on model performance in a low-resource Afrikaans question answering model. In 2025 Conference on Information Communications Technology and Society (ICTAS), pages 1–6. find
  46. Anastasia Nikiforova, Sergey Pletenev, Daria Sinitsyna, Semen Sorokin, Anastasia Lopukhina, and Nick Howell. 2020. Language models for cloze task answer generation in Russian. In Proceedings of the Second Workshop on Linguistic and Neurocognitive Resources, pages 28–37, Marseille, France. European Language Resources Association. find
  47. Byung-Doh Oh and Tal Linzen. 2025. To model human linguistic prediction, make LLMs less superhuman. arXiv preprint arXiv:2510.05141. ↗
  48. Jessica Ojo, Odunayo Ogundepo, Akintunde Oladipo, Kelechi Ogueji, Jimmy Lin, Pontus Stenetorp, and David Ifeoluwa Adelani. 2025. AfroBench: How good are large language models on African languages? In Findings of the Association for Computational Linguistics: ACL 2025, pages 19048–19095, Vienna, Austria. Association for Computational Linguistics. find
  49. Andrew M Olney. 2022. Assessing readability by filling cloze items with transformers. In International Conference on Artificial Intelligence in Education, pages 307–318. Springer. find
  50. JM O’Toole and RAR King. 2011. The deceptive mean: Conceptual scoring of cloze entries differentially advantages more able readers. Language Testing, 28(1):127–144. ↗
  51. PanSALB. 2021. The status of Afrikaans as an indigenous South African language. Accessed: 2026-02-16. find
  52. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534, Berlin, Germany. Association for Computational Linguistics. ↗
  53. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463–2473, Hong Kong, China. Association for Computational Linguistics. ↗
  54. Daniele Puccinelli, Silvia Demartini, and Pier Luigi Ferrari. 2021. Tackling Italian University assessment tests with transformer-based language models. In Proceedings of the Eighth Italian Conference on Computational Linguistics (CLiC-it 2021), pages 404–409, Milan, Italy. CEUR Workshop Proceedings. find
  55. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online. Association for Computational Linguistics. ↗
  56. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Technical Report. find
  57. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(1). ↗
  58. SADiLaR Language Resource Repository. Search results for Afrikaans. Accessed: 2026-02-17. find
  59. Emily Sadlier-Brown, Millie Lou, Miikka Silfverberg, and Carla Kam. 2024. How useful is context, actually? Comparing LLMs and humans on discourse marker prediction. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 231–241, Bangkok, Thailand. Association for Computational Linguistics.
  60. C. Shain, C. Meister, T. Pimentel, R. Cotterell, and R. Lévy. 2024. Large-scale evidence for logarithmic effects of word predictability on reading time. Proceedings of the National Academy of Sciences, 121. ↗
  61. Stanford NLP Group. 2024. Model performance – Stanza. Accessed: 19 February 2026. find
  62. W. L. Taylor. 1953. “Cloze procedure”: A new tool for measuring readability. Journalism Quarterly, 30:415–433. find
  63. Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment research. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5366–5377, Marseille, France. European Language Resources Association. find
  64. Sowmya Vajjala and Ivana Lučić. 2018. OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 297–304, New Orleans, Louisiana. Association for Computational Linguistics.
  65. Virtuele Instituut vir Afrikaans (VivA). Korpus-portaal: Oop (explore corpus). Accessed: 2026-02-17. find
  66. Rodrigo Wilkens, Patrick Watrin, Rémi Cardon, Alice Pintard, Isabelle Gribomont, and Thomas François. 2024. Exploring hybrid approaches to readability: experiments on the complementarity between linguistic features and transformers. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2316–2331, St. Julian’s, Malta. Association for Computational Linguistics. find
  67. Qizhe Xie, Guokun Lai, Zihang Dai, and Eduard Hovy. 2018. Large-scale cloze test dataset created by teachers. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2344–2356, Brussels, Belgium. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lotz-2026-getting-close-cloze-investigating-language/