📄 A Language Modeling Approach to Predicting Reading Difficulty

Summary

Traditional readability formulas such as Flesch-Kincaid assume clean prose of at least about 100 words with well-formed sentences. Web pages often violate these assumptions, which is a problem for a student-oriented search engine that should match results to reading ability. Collins-Thompson and Callan treat readability as a statistical language modeling task. They assembled a Web corpus of English pages that site authors had labeled with grade levels, and they trained one unigram language model per US grade (1–12). A passage is scored against every grade model with a modified multinomial naïve Bayes classifier, the "Smoothed Unigram" model. Its key modification is smoothing word probabilities across neighbouring grades instead of treating the grades as independent classes. The classifier does not depend on subject matter, needs little labeled data, and outputs a full distribution over grades. Correlation with labeled grade stayed between 0.63 and 0.79 on all test sets. It beat traditional variables on Web documents and on very short passages. After a small morphology change, it also transferred to French.

Key points

• English corpus: 550 Web documents across 12 American grade levels, with 448,715 tokens and 17,928 types, covering fiction, non-fiction, history, science and other subjects.
• Scoring: each passage gets the summed log-probability of its tokens under each grade model, assuming uniform priors and a length distribution that does not depend on grade. The grade with the highest likelihood is predicted.
• Smoothing:
  • Probabilities are smoothed across grades with Gaussian kernel regression, with a best kernel width of 2.5 grade levels.
  • Words unseen in all grade models back off to an estimate based on word length.
  • Test-data type coverage was about 90%.
• French: 189 Web pages at 5 difficulty levels (394,410 tokens, 27,632 unstemmed types). With 10-fold cross-validation, the model reached a mean correlation of 0.64, against 0.48 for type/token ratio.
• Comparison with traditional variables: type/token ratio and similar variables did best on commercial calibrated passages but were inconsistent on Web test sets. The Smoothed Unigram model performed well on every test set and was stronger on passages under 10 words.

Interesting ideas

• I1. Ordered classes as a continuum. Kernel-smoothing word frequencies across neighbouring grades lets each class borrow evidence from nearby classes. The authors say this multiplies a grade model's effective vocabulary by roughly five or more. The trick suits any ordinal labeling with sparse data per label, such as CEFR levels, time periods or age groups.
• I2. Relative rather than absolute probabilities. The useful signal is a word's trajectory across grades: when it first appears, whether its use rises or falls, and whether it peaks in a certain range. Absolute frequencies would mostly encode training-set topics. Feature design that focuses on such trajectories can help keep content separate from difficulty or style in other tasks.
• I3. Word lists as degenerate language models. Classic vocabulary-based measures such as Dale-Chall and Lexile can be read as simplified language models. Seeing them this way gives a principled path to generalize hand-built lexicons into probabilistic models.
• I4. Cheap transfer to an inflected language. Collapsing word forms into "type families" with crude stemming (plurals, gender endings, basic verb endings) was the only change needed for French. It took only a few hours of effort. This is relevant for morphologically rich languages like Ukrainian, where a light normalization step could stand in for heavier tooling.
• I5. Distribution-valued output. The model gives likelihoods over all grades rather than a single label. This supports ranking, confidence estimates and combining top predictions. The authors note that averaging the top two predictions can pull results toward the middle grades.