Elamur Research⌕
☰
home / papers / 2005 / Reading Level Assessment Using Support Vector Machines and Statistical Language Models

Reading Level Assessment Using Support Vector Machines and Statistical Language Models

Summary

The paper addresses automatic reading level estimation. The motivating use case is helping bilingual-education teachers find topical texts that are easy enough for English learners. Classic formulas such as Flesch-Kincaid and Lexile rely on sentence length, syllable counts or fixed word lists, so they miss real syntactic difficulty and penalise necessary topic vocabulary.

The authors train on the Weekly Reader corpus, which has roughly 2,400 articles from grade 2–5 editions. They add two supplemental corpora of adult texts paired with simplified versions: Encyclopedia Britannica/Britannica Elementary and CNN/abridged CNN.

They build one detector per grade. Each detector uses n-gram language model perplexities as features, alongside parse-based features from the Charniak parser and traditional measures. An SVM combines all of these. Trigram models outperform bigram and unigram models, and the SVM combination gives the best results overall.

Compared with Flesch-Kincaid and Lexile, the SVM misclassifies far fewer articles by more than one grade. It also behaves sensibly on out-of-domain Washington Post texts. This shows that cheap, corpus-trainable LM features capture readability better than static formulas.

Key points

  • Data:

    • Weekly Reader grades 2/3/4/5 contain 351/589/766/691 articles respectively.
    • Supplemental corpora: Britannica (115 articles) with Britannica Elementary (115), and CNN (111) with CNN Abridged (111).

  • LM setup:

    • Models were trained with the SRI Language Modeling Toolkit using modified Kneser-Ney smoothing.
    • Vocabulary was selected by information gain.
    • Perplexity worked better than the likelihood ratio as an SVM feature.

  • Detection performance: Equal error rates on the DET curves range from about 12% to 25% across grades. At the chosen operating points, test-set precision/recall were:

    GradePrecisionRecall
    238%61%
    338%87%
    470%60%
    575%79%

  • Comparison with traditional measures: Share of articles misclassified by more than one grade:

    GradeSVMLexileFlesch-Kincaid
    25.5%33%78%
    33.3%27%67%
    413%26%74%
    521%24%59%

  • Out-of-domain check:

    • Of 10 "Kidspost" articles (aimed at grades 3–8), the classifier labelled four as grade 4 and seven as grade 5, with one article receiving both labels.
    • Adult Washington Post articles were assigned the highest grade available.

Interesting ideas

  • I1. One yes/no detector per level instead of a single multi-class ranker. This matches how a teacher actually searches ("give me grade-3 texts"). Each detector's threshold can be tuned independently, which suits retrieval-style applications.
  • I2. Language model perplexities as features. Training a separate n-gram model per level and feeding its perplexities into a discriminative classifier gives a cheap stand-in for both lexical and syntactic difficulty. This transfers easily to languages or domains without readability formulas, such as Ukrainian.
  • I3. Parallel original/simplified corpora as extra training data. Pairs like Britannica vs. Britannica Elementary, or CNN vs. abridged CNN, can train coarse "easy vs. hard" models. Their outputs then serve as features for finer classification, a useful trick when fine-grained labels are scarce.
  • I4. An error metric that reflects ordered labels. Counting predictions that are off by more than one level penalises large ordinal mistakes. It is more informative than plain accuracy for graded scales.
  • I5. Operating points chosen on a development set via DET curves. Picking thresholds this way, and reporting precision/recall at those points, makes the tradeoff explicit. It shows clearly why middle levels (grades 3–4) are harder than the extremes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

129 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 116 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1219840.1219905

@inproceedings{schwarm2005reading,
  title = {{Reading Level Assessment Using Support Vector Machines and Statistical Language Models}},
  author = {Schwarm, Sarah and Ostendorf, Mari},
  booktitle = {Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05)},
  year = {2005},
  doi = {10.3115/1219840.1219905},
  url = {https://doi.org/10.3115/1219840.1219905}
}

References

27 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Barzilay and N. Elhadad. Sentence alignment for monolingual comparable corpora. In Proc. of EMNLP, pages 25–32, 2003. find our card
  2. C. Boulis and M. Ostendorf. Text classification by augmenting the bag-of-words representation with redundancy-compensated bigrams. Workshop on Feature Selection in Data Mining, in conjunction with SIAM conference on Data Mining, 2005. find
  3. P. Bylsma, L. Ireland, and H. Malagon. Educating English Language Learners in Washington State. Office of the Superintendent of Public Instruction, Olympia, WA, 2003. find
  4. J.S. Chall and E. Dale. Readability revisited: the new Dale-Chall readability formula. Brookline Books, Cambridge, Mass., 1995. find
  5. E. Charniak. A maximum-entropy-inspired parser. In Proc. of NAACL, pages 132–139, 2000. find our card
  6. C. Chelba and F. Jelinek. Structured Language Modeling. Computer Speech and Language, 14(4):283-332, 2000. find
  7. S. Chen and J. Goodman. An empirical study of smoothing techniques for language modeling. Computer Speech and Language, 13(4):359–393, 1999. find
  8. K. Collins-Thompson and J. Callan. A language modeling approach to predicting reading difficulty. In Proc. of HLT/NAACL, pages 193–200, 2004. find
  9. R. Gunning. The technique of clear writing. McGraw-Hill, New York, 1952. find
  10. C.-W. Hsu et al. A practical guide to support vector classification. http://www.csie.ntu.edu.tw/˜cjlin/papers/guide/guide.pdf, 2003. Accessed 11/2004. ↗
  11. T. Joachims. Text categorization with support vector machines: learning with many relevant features. In Proc. of the European Conference on Machine Learning, pages 137–142, 1998a. find our card
  12. T. Joachims. Making large-scale support vector machine learning practical. In Advances in Kernel Methods: Support Vector Machines. B. Schölkopf, C. Burges, A. Smola, eds. MIT Press, Cambridge, MA, 1998b. find
  13. J.P. Kincaid, Jr., R.P. Fishburne, R.L. Rodgers, and B.S. Chisson. Derivation of new readability formulas for Navy enlisted personnel. Research Branch Report 8-75, U.S. Naval Air Station, Memphis, 1975. find
  14. Y.-B. Lee and S.H. Myaeng. Text genre classification with genre-revealing and subject-revealing features. In Proc. of SIGIR, pages 145–150, 2002. find
  15. The Lexile framework for reading. http://www.lexile.com, 2005. Accessed April 15, 2005. ↗
  16. A. Martin, G. Doddington, T. Kamm, M. Ordowski, and M. Przybocki. The DET curve in assessment of detection task performance. Proc. of Eurospeech, v. 4, pp. 1895-1898, 1997. find
  17. A. Ratnaparkhi. A maximum entropy part-of-speech tagger. In Proc. of EMNLP, pages 133–141, 1996. find
  18. B. Roark. Probabilistic top-down parsing and language modeling. Computational Linguistics, 27(2):249-276, 2001. find our card
  19. L. Si and J.P. Callan. A statistical model for scientific readability. In Proc. of CIKM, pages 574–576, 2001. find
  20. A.J. Stenner. Measuring reading comprehension with the Lexile framework. Presented at the Fourth North American Conference on Adolescent/Adult Literacy, 1996. find
  21. A. Stolcke. SRILM - an extensible language modeling toolkit. Proc. ICSLP, v. 2, pp. 901-904, 2002. find
  22. U.S. Department of Education, National Center for Educational Statistics. The condition of education. http://nces.ed.gov/programs/coe/2003/section1/indicator04.asp, 2003. Accessed June 18, 2004. ↗
  23. U.S. Department of Education, National Center for Educational Statistics. NCES fast facts: Bilingual education/Limited English Proficient students. http://nces.ed.gov/fastfacts/display.asp?id=96, 2003. Accessed June 18, 2004. ↗
  24. V. Vapnik. The Nature of Statistical Learning Theory. Springer, New York, 1995. find
  25. The Washington Post. http://www.washingtonpost.com, 2005. Accessed April 20, 2005. ↗
  26. Weekly Reader. http://www.weeklyreader.com, 2004. Accessed July, 2004. ↗
  27. Western/Pacific Literacy Network / Literacyworks. CNN SF learning resources. http://literacynet.org/cnnsf/, 2004. Accessed June 15, 2004. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schwarm-2005-reading-level-assessment-support-vector/