Elamur Research⌕
☰
home / papers / 2023 / Traditional Readability Formulas Compared for English

Traditional Readability Formulas Compared for English

Paper · 2023 · Bruce W. Lee and Jason Hyung-Jong Lee

Summary

Many NLP studies still score text difficulty with mid-20th-century readability equations. Those equations were fitted on military and technical material, and their linguistic coverage is narrow. The authors refit the coefficients of five classic formulas (FKGL, Fog Index, SMOG, Coleman-Liau, ARI) on 69 story excerpts from Appendix B of the U.S. Common Core State Standards (CCB). The fitting uses non-linear least squares, with grade bands converted to point targets.

They also build NERF, a new formula drawing on lexico-semantic and syntactic signals such as word familiarity and age-of-acquisition. Its features were screened across several corpora: WeeBit, Cambridge English, the Korean ELT curriculum corpus, OneStopEnglish, and Newsela. All formulas are tested on CCB, on passage- and sentence-level simplification pairs, and on patient medical leaflets. A Python tool is released alongside.

Results:

  • Recalibration clearly improves agreement with the U.S. curriculum.
  • Ranking simplified passages works well.
  • Ranking sentences mostly fails.
  • NERF outperforms the other formulas on paired comparisons and curbs the inflated difficulty scores the old formulas give medical texts.

Key points

  • The calibration data is only CCB story texts: 69 items. Its range labels (K1, K2-3, …, college) were mapped to K1, K2.5, K4.5, K7, K9.5 and K12.
  • Refitting raised FKGL's r² on CCB from about −0.04 to 0.44 and its Pearson r from 0.57 to 0.67. ARI moved from −0.53 to 0.43 r². Coleman-Liau was strongest both before and after (r² 0.43 → 0.48).
  • On the OneStopEnglish pairs, the original formulas stayed at or below 95% accuracy. Most improved after recalibration, and NERF did better still. On the Newsela five-way passage ranking, every formula scored 100%, which suggests that set is too easy to discriminate.
  • On ASSET (ranking 11 versions of a sentence across 360 items), only SMOG got any right: 89 originally and 37 after recalibration. The others scored zero.
  • On 18 American Urogynecologic Society patient leaflets, NERF kept scores from inflating the way syllable-based formulas do. The NIH target for such leaflets is grade 6 or below.

Interesting ideas

  • I1. Keep a tiny, interpretable model form, such as a two- or three-term equation, and refit only its constants on a small trusted gold set. This gains accuracy without losing transparency.
  • I2. When gold labels are grade ranges, replace each range with a representative point (e.g., K2-3 → 2.5). This lets ordinary regression fit a coarse ordinal scale.
  • I3. Score evaluation as ranking several versions of the same content rather than predicting absolute values. If every system hits 100%, read that as a sign the benchmark is saturated, not as success.
  • I4. Estimate word difficulty from familiarity and age-of-acquisition norms instead of length. Unknown specialist terms stay neutral, and repeated terms are not counted as extra difficulty.
  • I5. A measure calibrated on passages can collapse on single sentences, so the unit of evaluation must match the unit of calibration.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Mova could expose a text-difficulty score per school grade whose few coefficients are fitted to graded course readings, so every score can be traced to explicit counts.
  • I2 (learn): the same trick could turn banded difficulty or confidence tiers from LLM teachers into numeric training targets.
  • I3 (back): outputs of the text → graph → text compressor could be checked by whether simpler regenerations are ranked as simpler, with a saturated pair set treated as a warning.
  • I4 (tokens): age-of-acquisition and frequency values could be attached to lemmas in crate en, giving an explainable lexical-difficulty feature that does not punish long but common words.
  • I5 (answer): any difficulty or complexity check used in school tests should be validated separately at sentence level and passage level before being trusted at either.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Bruce W. Lee and Jason Hyung-Jong Lee. 2023. Traditional Readability Formulas Compared for English. arXiv:2301.02975.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.02975 · arXiv 2301.02975

@misc{lee2023traditional,
  title = {{Traditional Readability Formulas Compared for English}},
  author = {Bruce W. Lee and Jason Hyung-Jong Lee},
  year = {2023},
  doi = {10.48550/arxiv.2301.02975},
  eprint = {2301.02975},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.02975}
}

References

55 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandra Aluisio, Lucia Specia, Caroline Gasperin, and Carolina Scarton. 2010. Readability assessment for text simplification. In Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–9.
  2. Fernando Alva-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, Benoît Sagot, and Lucia Specia. 2020. Asset: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. arXiv preprint arXiv:2005.00481. ↗
  3. Matthew Bange, Eric Huh, Sherwin A Novin, Ferdinand K Hui, and Paul H Yi. 2019. Readability of patient education materials from radiologyinfo. org: has there been progress over the past 5 years? American Journal of Roentgenology, 213(4):875–879. ↗
  4. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.". find
  5. Marc Brysbaert, Matthias Buchmeier, Markus Conrad, Arthur M Jacobs, Jens Bölte, and Andrea Böhl. 2011. The word frequency effect. Experimental psychology. ↗
  6. Marc Brysbaert and Boris New. 2009. Moving beyond kučera and francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for american english. Behavior research methods, 41(4):977–990. ↗
  7. Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283. ↗
  8. Kevyn Collins-Thompson. 2014. Computational assessment of text readability: A survey of current and future research. ITL-International Journal of Applied Linguistics, 165(2):97–135. find
  9. Edgar Dale and Jeanne S Chall. 1948. A formula for predicting readability: Instructions. Educational research bulletin, pages 37–54. find
  10. Murphy Odo Dennis. 2018. A comparison of readability and understandability in second language acquisition textbooks for pre-service efl teachers. Journal of Asia TEFL, 15(3):750–765. ↗
  11. William H DuBay. 2004. The principles of readability. Online Submission. find
  12. Lijun Feng, Martin Jansche, Matt Huenerfauth, and Noémie Elhadad. 2010. A comparison of features for automatic readability assessment.
  13. Laura Gaeta, Edward Garcia, and Valeria Gonzalez. 2021. Readability and suitability of spanish-language hearing aid user guides. American Journal of Audiology, 30(2):452–457. ↗
  14. Robert Gunning et al. 1952. Technique of clear writing. find
  15. Jasmine Haller, Zachary Keller, Susan Barr, Kristie Hadden, and Sallie S Oliphant. 2019. Assessing readability: are urogynecologic patient education materials at an appropriate reading level? Female pelvic medicine & reconstructive surgery, 25(2):139–144. ↗
  16. David R Hansberry, Michael D’Angelo, Michael D White, Arpan V Prabhu, Mougnyan Cox, Nitin Agarwal, and Sandeep Deshmukh. 2018. Quantitative analysis of the level of readability of online emergency radiology-based patient education resources. Emergency radiology, 25(2):147–152. ↗
  17. Matthew Honnibal and Mark Johnson. 2015. An improved non-monotonic transition system for dependency parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1373–1378, Lisbon, Portugal. Association for Computational Linguistics. ↗
  18. Pradeep Joseph, Nicole A Silva, Anil Nanda, and Gaurav Gupta. 2020. Evaluating the readability of online patient education materials for trigeminal neuralgia. World Neurosurgery, 144:e934–e938. ↗
  19. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research Branch. ↗
  20. Elizabeth Kiwanuka, Raman Mehrzad, Adnan Prsic, and Daniel Kwan. 2017. Online patient resources for gender affirmation surgery: an analysis of readability. Annals of plastic surgery, 79(4):329–333. ↗
  21. George R Klare. 2000. The measurement of readability: useful information for communicators. ACM Journal of Computer Documentation (JCD), 24(3):107–121. ↗
  22. Jennifer Kue, Dori L Klemanski, and Kristine K Browning. 2021. Evaluating readability scores of treatment summaries and cancer survivorship care plans. JCO Oncology Practice, pages OP–20. ↗
  23. Victor Kuperman, Hans Stadthagen-Gonzalez, and Marc Brysbaert. 2012. Age-of-acquisition ratings for 30,000 english words. Behavior research methods, 44(4):978–990. ↗
  24. Bruce W Lee, Yoo Sung Jang, and Jason Lee. 2021. Pushing on text readability assessment: A transformer meets handcrafted linguistic features. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10669– 10686.
  25. Bruce W. Lee and Jason Lee. 2020a. LXPER index 2.0: Improving text readability assessment model for L2 English students in Korea. In Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications, pages 20– 24, Suzhou, China. Association for Computational Linguistics.
  26. Bruce W. Lee and Jason Hyung-Jong Lee. 2020b. Lxper index: A curriculum-specific text readability assessment model for efl students in korea. International Journal of Advanced Computer Science and Applications, 11(8). ↗
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  28. Bertha A Lively and Sidney L Pressey. 1923. A method for measuring the vocabulary burden of textbooks. Educational administration and supervision, 9(7):389–398. find
  29. David Malvern and Brian Richards. 2012. Measures of lexical richness. The encyclopedia of applied linguistics. ↗
  30. G Harry Mc Laughlin. 1969. Smog grading-a new readability formula. Journal of reading, 12(8):639–646. find
  31. Changping Meng, Muhao Chen, Jie Mao, and Jennifer Neville. 2020. Readnet: A hierarchical transformer framework for web article readability analysis. Advances in Information Retrieval, 12035:33. find
  32. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  33. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  34. Lauren E Powell, Emily S Andersen, and Andrea L Pozez. 2020. Assessing readability of patient education materials on breast reconstruction by major us academic institutions. Plastic and Reconstructive Surgery–Global Open, 8(9S):127–128. ↗
  35. Carolina Scarton and Lucia Specia. 2018. Learning simplifications for specific target audiences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 712–718.
  36. H. Andrew Schwartz, Masoud Rouhizadeh, Michael Bishop, Philip Tetlock, Barbara Mellers, and Lyle Ungar. 2017. Assessing objective recommendation quality through political forecasting. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2348–2357, Copenhagen, Denmark. Association for Computational Linguistics. find
  37. Matthew Shardlow and Raheel Nawaz. 2019. Neural text simplification of clinical letters with a domain specific phrase table. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 380–389. find
  38. Edgar A Smith and RJ Senter. 1967. Automated readability index. AMRL-TR. Aerospace Medical Research Laboratories (US), pages 1–14. find
  39. T Szmuda, C Özdemir, S Ali, A Singh, MT Syed, and P Słoniewski. 2020. Readability of online patient education material for the novel coronavirus disease (covid-19): a cross-sectional health literacy study. Public Health, 185:21–25. ↗
  40. Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. 2020. Authorship attribution for neural text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8384–8395, Online. Association for Computational Linguistics.
  41. Sowmya Vajjala and Ivana Lučić. 2018. Onestopenglish corpus: A new corpus for automatic readability assessment and text simplification. In Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications, pages 297–304.
  42. Sowmya Vajjala and Detmar Meurers. 2012. On improving the accuracy of readability classification using insights from second language acquisition. In Proceedings of the seventh workshop on building educational applications using NLP, pages 163–173.
  43. Courtney van Ballegooie and Peter Hoang. 2021. Assessment of the readability of online patient education material from major geriatric associations. Journal of the American Geriatrics Society, 69(4):1051– 1056. ↗
  44. N Verhelst, Piet Van Avermaet, S Takala, N Figueras, and B North. 2001. Common European Framework of Reference for Languages: learning, teaching, assessment. Cambridge University Press. find
  45. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. 2020. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261–272.
  46. Orion Weller, Jordan Hildebrandt, Ilya Reznik, Christopher Challis, E Shannon Tass, Quinn Snell, and Kevin Seppi. 2020. You don’t have time to read this: An exploration of document reading time prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1789– 1794. find
  47. Wes McKinney. 2010. Data Structures for Statistical Computing in Python. In Proceedings of the 9th Python in Science Conference, pages 56 – 61. find
  48. Danny TY Wu, David A Hanauer, Qiaozhu Mei, Patricia M Clark, Lawrence C An, Jianbo Lei, Joshua Proulx, Qing Zeng-Treitler, and Kai Zheng. 2013. Applying multiple methods to assess the readability of a large corpus of medical documents. Studies in health technology and informatics, 192:647. find
  49. Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2016. Text readability assessment for second language learners. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 12–22.
  50. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297. ↗
  51. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415. ↗
  52. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32. ↗
  53. Yu Zhang, Houquan Zhou, and Zhenghua Li. 2020. Fast and accurate neural CRF constituency parsing. In Proceedings of IJCAI, pages 4046–4053. find
  54. Jiaping Zheng and Hong Yu. 2017. Readability formulas and user perceptions of electronic health records difficulty: a corpus study. Journal of medical Internet research, 19(3):e59. ↗
  55. Shixiang Zhou, Heejin Jeong, and Paul A Green. 2017. How consistent are the best-known readability equations in estimating the readability of design standards? IEEE Transactions on Professional Communication, 60(1):97–111. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lee-2023-traditional-readability-formulas-compared-english/