Elamur Research⌕
☰
home / papers / 2018 / Characterizing the Influence of Features on Reading Difficulty Estimation for Non-native Readers

Characterizing the Influence of Features on Reading Difficulty Estimation for Non-native Readers

Paper · 2018 · Yi-Ting Huang, Meng Chang Chen and Yeali S. Sun

Summary

Most readability formulas are tuned for native speakers, but people learning English as a second language acquire vocabulary and grammar gradually and in the order their courses teach them. The authors therefore build a reading-difficulty estimator aimed at ESL readers. They fit a linear regression that predicts the grade level of English texts from a broad set of candidate features. These include classic lexical and syntactic measures and several features new to this setting:

  • age of word acquisition, taken from graded vocabulary indices;
  • age of grammar acquisition, based on grammar patterns collected from textbooks;
  • word-sense information from WordNet;
  • coreference links between sentences.

Features are added by forward selection, and competing models are compared with the Bayesian Information Criterion (BIC). The evaluation corpus is a set of English textbooks for Taiwanese high-school students. The best model needs only three predictors: word count, age of word acquisition and parse-tree height. It beats the first-language readability estimators it was compared against. This matters for graded search, learner-oriented summarization and textbook authoring, where content must match what a learner has already been taught.

Key points

  • The target audience is ESL readers. The authors argue that L1 formulas such as Dale-Chall, Flesch-Kincaid and Lexile depend on native-corpus word frequency and transfer badly to learners.
  • There are four new feature families: vocabulary acquisition age from grading indices, 44 textbook-derived grammar patterns, WordNet word senses, and coreference relations.
  • The model is a linear regression with forward feature selection, scored by BIC. Individual features were also assessed one by one for accuracy.
  • The data are ESL textbooks used in Taiwanese high schools.
  • The winning combination is number of words + age of word acquisition + parsing-tree height. Age of word acquisition is the most influential single signal. The model outperforms L1 baselines, and the paper also discusses the Heilman et al. (2007) approach.

Interesting ideas

  • I1. Measure word difficulty by when a curriculum introduces a word, not by how often it appears in a native corpus, because learner exposure follows the textbook rather than general usage.
  • I2. Treat grammar constructions as items that are acquired at a known grade, and detect them as patterns over parse trees. A text's difficulty then reflects the latest construction a learner must already know.
  • I3. Use forward selection with BIC so that only a few interpretable predictors survive, instead of a large opaque feature set.
  • I4. Treat polysemy as a difficulty factor: a common word used in a rarer sense can be harder than its frequency suggests.
  • I5. Treat cross-sentence reference resolution as part of what makes a text hard, not only word and sentence properties.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-course readers could tag each lemma with the grade at which it is taught. This would let Mova pick or explain test passages at the right level for a student.
  • I2 (tree): The 44-pattern idea maps naturally onto UD subtree queries (relative clauses, passives, conditionals). Each detected construction would be explainable and linked to a grade.
  • I3 (learn): The same procedure could prune features in Mova's evaluation or repair-rule scoring. It would keep each retained signal justified by a penalized fit rather than added by habit.
  • I4 (world): With WordNet senses already in the compiled knowledge, Mova could flag rare-sense uses in a text and explain why a sentence is hard to understand.
  • I5 (frames): The density and distance of coreference chains found by the coreference step could serve as an explainable text-difficulty measure for graded reading tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

Cite

Cite as Yi-Ting Huang, Meng Chang Chen and Yeali S. Sun. 2018. Characterizing the Influence of Features on Reading Difficulty Estimation for Non-native Readers. arXiv:1808.09718.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1808.09718 · arXiv 1808.09718

@misc{huang2018characterizing,
  title = {{Characterizing the Influence of Features on Reading Difficulty Estimation for Non-native Readers}},
  author = {Yi-Ting Huang, Meng Chang Chen and Yeali S. Sun},
  year = {2018},
  doi = {10.48550/arxiv.1808.09718},
  eprint = {1808.09718},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1808.09718}
}

References

38 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agrawal, R., Gollapudi, S., Kannan, A., and Kenthapadi, K. (2011). Identifying Enrichment Candidates in Textbooks. Proceedings of International World Wide Web Conference (WWW2011) (pp. 483–492) find
  2. Bates, E. (2003). On the nature and nurture of language. Retrieved November 24, 2011, from http://crl.ucsd.edu/bates/papers/pdf/bates-inpress.pdf ↗
  3. Barzilay, R. and Lapata, M. (2008). Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1), 1-34. ↗
  4. Bouckaert, R. R., Frank, E., Hall, M. A., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I. H. and Sonnenburg, S. (2010). WEKA-experiences With a Java Open-source Project. Journal of Machine Learning Research, 11, 2533-2541. ↗
  5. Coleman, M. and Liau, T. L. (1975). A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283–284. ↗
  6. Collins-Thompson, K. and Callan, J. (2004). A Language Modeling Approach to Predicting Reading Difficulty. Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL2004)
  7. Collins-Thompson, K., Bennett, P. N., White, R. W., Chica, S. Sontag, D. (2011). Personalizing Web Search Results by Reading Level. Proceedings of CIKM2011 find
  8. Curtis, A., Romney, M. (2006). Color, Race, And English Language Teaching: Shades of Meaning. Lawrence Erlbaum Associates. find
  9. Dale, E. and Chall, J. S. (1948). A Formula for Predicting Readability. Educational Research Bulletin, Vol. 27(1). find
  10. EL-Manzalawy, Y. and Honavar, V. (2005). {WLSVM}: Integrating LibSVM into Weka Environment. the proposed estimation into the AutoQuiz Retrieved November 24, 2011, from http://www.cs.iastate.edu/~yasser/wlsvm/English_language. (n.d.). In Wikipedia. http://en.wikipedia.org/wiki/English_language ↗
  11. Feng, L., Jansche, M., Huenerfauth, M., and Elhadad, N. (2010). A Comparison of Features for Automatic Readability Assessment. Proceedings of International Conference on Computational Linguistics (Coling 2010) (pp. 276-284).
  12. Flesch, R. (1948). A new readability yardstick. Journal of applied psychology, 32(3), 221-233. ↗
  13. Gunning, R. (1952). The technique of clear writing. find
  14. Heilman, M., Collins-Thompson, K., Callan, J., and Eskenazi, M. (2007). Combining Lexical and Grammatical Features to Improve Readability Measures for First and Second Language Texts. Proceedings of the Human Language Technology Conference (HLT-NAACL2007) (pp. 460-467). find
  15. Heilman, M. , Collins-Thompson, K. and Eskenazi, M. (2008). An Analysis of Statistical Models and Features for Reading Difficulty Prediction. Proceedings of the Third ACL Workshop on Innovative Use of NLP for Building Educational Applications, (pp. 71–79). find
  16. Ho, H. and Huong, C. (2011). A Multiple Aspects Quantitative Indicator for Ability of English Vocabulary: Vocabulary Quotient. Journal of Educational Technology Development and Exchange, 4(1), 15-26. ↗
  17. Huang, Y. T., Chang, H. P., Sun, Y., and Chen, M. C. (2011). A Robust Estimation Scheme of Reading Difficulty for Second Language Learners. Processing of International Conference Advanced Learning Technology (ICALT2011) (pp. 58-62). Kate, R. J., Luo, X., Patwardhan, S., Franz, M., Florian, R., Mooney, R. J., Roukos, S., Welty, C. find
  18. (2010). Learning to Predict Readability using Diverse Lingustic Features. Proceedings of the 23 rd Internation Conference on Computational Lingustics, pages 546-554. find
  19. Kidwell, P., Lebanon, G., and Collins-Thompson, K. (2009). Statistical estimation of word acquisition with application to readability prediction. Proceedings of Empirical Methods in Natural Language Processing (EMNLP2009) (pp. 900-909). Singapore: Association for Computational Linguistics. ↗
  20. Kincaid, J. Peter; Fishburne, Lieutenant Robert P., Jr.; Rogers, Richard L.; Chissom, Brad S. (1975). Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Branch Report. Virginia: National Technical Information Service. ↗
  21. Kireyev, K. and Landauer, T. K. (2011). Word Maturity: Computational Modeling of Word Knowledge. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (HLT2011) (pp. 299-308). find
  22. Klein, D. and Manning, C. D. (2003). Accurate Unlexicalized Parsing. Proceedings of the 41st Meeting of the Association for Computational Linguistics (ACL03) (pp. 423-430).
  23. Levy, R. and Andrew, G. (2006). Tregex and Tsurgeon: tools for querying and manipulating tree data Retrieved November 23, 2004, from structures. Proceedings of International Conference on Language Resources and Evaluation (LREC 2006). find
  24. Lou, B and Guy, A. (1998). The BNC handbook: exploring the British National Corpus. Edinburgh: Edinburgh University Press. find
  25. McLaughlin, G. (1969). SMOG grading: A new readability formula. Journal of Reading, 12(8), 639–646. find
  26. Miller, G. A., Beckwith, R., Fellbaum, C. D., Gross, D., Miller, K. (1990). WordNet: An online lexical database. International Journal of Lexicography, 3(4), 235–244. find
  27. Pitler, E. and Nenkova, A. (2008). Revisiting Readability: A Unified Framework for Predicting Text Quality. Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP2008) (pp. 186-195).
  28. Schwarm, S. and Ostendorf, M. (2005). Reading Level Assessment Using Support Vector Machines and Statistical Language Models. Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (ACL05) (pp. 523-530).
  29. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6 (2), 461–464. ↗
  30. Shevade, S. K., Keerthi, S. S., Bhattacharyya, C., Murthy, K. R. K. (2000). Improvements to the SMO algorithm for SVM regression. IEEE Transactions on Neural Networks, 11(5), 1188-1193. ↗
  31. Stenner, A. J. (1996). Measuring Reading Comprehension with the Lexile Framework. Fourth North American Conference on Adolescent/Adult Literacy. find
  32. Tanaka-Ishii, K., Tezuka, S., Terada, T. (2010). Sorting Texts by Readability. Computational Linguistics, 36(2), 203-227. ↗
  33. Wan, X., Li, H., & Xiao, J. (2010). EUSUM: Extracting Easy-to-Understand English Summaries for Non-Native Readers. Proceedings of the 33th Annual International ACM SIGIR Conference ( SIGIR 2010) (pp. 491-498). find
  34. Wilson, M. (2003). MRC Psycholinguistic Database: Machine Usable Dictionary, Version 2.00. find
  35. Retrieved November 24, 2011, from http://www.psy.uwa.edu.au/MRCDataBase/MDWilsonMRC.pdf ↗
  36. Wu, R. Y. F., and Liao, C. H. Y. (2010). Establishing a Common Score Scale for the GEPT Elementary, Intermediate, and High-Intermediate Level Listening and Reading Tests. Proceedings of International Conference on English Language Teaching and Testing. Taipei: Language Training and Testing Center. find
  37. Yamada, J. (2004). An L1-script-transfer-effect fallacy: a rejoinder to Wang et al. (2003). Cognition, 93, 127-132. ↗
  38. Zhang, D., Jiang, Q. and Li, X. (2005). Application of Neural Networks in Financial Data Mining. World Academy of Science, Engineering and Technology, 1, 136–139. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2018-characterizing-influence-features-reading-difficulty/