Elamur Research⌕
☰
home / papers / 2010 / A Comparison of Features for Automatic Readability Assessment

A Comparison of Features for Automatic Readability Assessment

Summary

The paper asks which kinds of linguistic evidence best predict how hard a text is for young readers. The authors cast readability as a classification problem: given an article, assign one of the elementary grade levels 2–5. Their data come from the Weekly Reader archive. After removing puzzle and quiz pages, 1433 of 1629 labeled articles remain. They build a large feature inventory covering five families: shallow features, language-model scores, POS-based counts, parse-based syntactic counts, and discourse features. The discourse family includes entity density, lexical chains, entity grids and a new set of coreference-inference measures. Classifiers built with LIBSVM and Weka's Logistic Regression are compared under 10-fold cross-validation. Language models trained on in-domain text are the strongest predictors. Noun-based POS features and entity density are each useful, but they largely overlap. Plain average sentence length outperforms the individual syntactic measures, and discourse features add little at these low grade levels. A greedy, group-wise selected feature set reaches about 74% accuracy, well above a replicated prior state of the art (about 63%).

Key points

  • Corpus: Weekly Reader articles for grades 2–5 (174 / 289 / 428 / 542 documents after filtering). Mean sentence length grows steadily with grade, from about 9.5 to about 15.3 words.
  • Toolchain:
    • OpenNLP for named entities and coreference.
    • Charniak's parser for nouns and syntax.
    • Galley and McKeown's lexical chainer.
    • Brown Coherence Toolkit v0.2, which yields 16 entity-transition probability features.
  • Baselines:
    • Always guessing grade 5 gives 37.8% accuracy.
    • Rounding the Flesch-Kincaid Grade Level score matches the label for only 20 of 1433 texts (1.4%).
    • The same Flesch-Kincaid score used as a logistic-regression input exceeds 50%.
  • LIBSVM results:
Feature setNumber of featuresAccuracy
Replicated Schwarm & Ostendorf features2563.18%
All features27372.21%
WekaFS2870.06%
AddOneBest12274.01%
  • Shallow features: average sentence length dominates, and syllable-based features do much worse. SBAR counts were not helpful, while verb-phrase features tracked complexity better than other phrase types.

Interesting ideas

  • I1. Learn the weights of classic formulas instead of trusting them. The same two Flesch-Kincaid ingredients jumped from near-useless to above 50% once a classifier learned their weights. Before discarding a hand-tuned metric for a new domain or language, it is worth refitting its inputs on that data.
  • I2. Select features group by group. The authors' greedy selection adds whole feature families at a time, keeping the family that helps most. It beat both the naive use of all features and Weka's automatic filter. This makes ablations easier to interpret and is cheap whenever features come in natural groups.
  • I3. Re-run the prior work on your own data. Because the previous study used a different split and toolkit, the authors re-implemented its features on their corpus with the same cross-validation setup. This gives a fair head-to-head comparison rather than comparing numbers taken from different papers.
  • I4. Train language models on the target corpus. LMs trained on the target corpus beat LMs trained on unrelated text. For a new language, building small in-domain LMs per difficulty level is likely a strong first signal.
  • I5. Check whether rich features beat cheap proxies. Entity density largely duplicated the signal from noun counts, and sentence length beat parser-derived measures. Testing whether an expensive feature only restates a cheap one can save heavy preprocessing, such as parsing or coreference, when building tools for less-resourced languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Discourse analysis · Coreference resolution

Cited by

82 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 77 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{feng2010comparison,
  title = {{A Comparison of Features for Automatic Readability Assessment}},
  author = {Feng, Lijun and Jansche, Martin and Huenerfauth, Matt and Elhadad, Noémie},
  booktitle = {Coling 2010: Posters},
  year = {2010},
  url = {https://aclanthology.org/C10-2032/}
}

References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandra Aluisio, Lucia Specia, Caroline Gasperin, and Carolina Scarton. 2010. Readability assessment for text simplification. In NAACL-HLT 2010: The 5th Workshop on Innovative Use of NLP for Building Educational Applications. find our card
  2. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1–34. find our card
  3. Chih-Chung Chang and Chih-Jen Lin. 2001. LIB-SVM: A Library for Support Vector Machines. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm. ↗
  4. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st Conference of the North American Chapter of the ACL, pages 132–139. find our card
  5. Kevyn Collins-Thompson and Jamie Callan. 2004. A language modeling approach to predicting reading difficulty. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL 2004). find
  6. Micha Elsner, Joseph Austerweil, and Eugene Charniak. 2007. A unified local and global model for discourse coherence. In Proceedings of the Conference on Human Language Technology and North American chapter of the Association for Computational Linguistics (HLT-NAACL 2007). find our card
  7. Lijun Feng, Noémie Elhadad, and Matt Huenerfauth. 2009. Cognitively motivated features for readability assessment. In The 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2009). find
  8. Rudolf Flesch. 1979. How to write plain English. Harper and Brothers, New York. find
  9. Michel Galley and Kathleen McKeown. 2003. Improving word sense disambiguation in lexical chaining. In Proceedings of the 18th International Joint Conference on Artificial Intelligence. find
  10. Robert Gunning. 1952. The Technique of Clear Writing. McGraw-Hill. find
  11. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. SIGKDD Explorations, 11(1):10–18. find
  12. Michael J. Heilman, Kevyn Collins-Thompson, Jamie Callan, and Maxine Eskenazi. 2007. Combining lexical and grammatical features to improve readability measures for first and second language texts. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association fo… find
  13. Michael J. Heilman, Kevyn Collins-Thompson, and Maxine Eskenazi. 2008. An analysis of statistical models and features for reading difficulty prediction. In ACL 2008: The 3rd Workshop on Innovative Use of NLP for Building Educational Applications. find
  14. Mirella Lapata and Regina Barzilay. 2005. Automatic evaluation of text coherence: Models and representations. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’05), pages 1085–1090. find
  15. Gondy Leroy, Stephen Helmreich, James R. Cowie, Trudi Miller, and Wei Zheng. 2008. Evaluating online health information: Beyond readability formulas. In AMIA 2008 Symposium Proceedings. find
  16. G. Harry McLaughlin. 1969. Smog grading a new readability formula. Journal of Reading, 12(8):639–646. find
  17. Sarah E. Petersen and Mari Ostendorf. 2006. A machine learning approach to reading level assessment. Technical report, University of Washington CSE Technical Report. find
  18. Sarah E. Petersen and Mari Ostendorf. 2009. A machine learning approach to reading level assessment. Computer Speech and Language, 23:89– 106. find
  19. Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. find our card
  20. R. Prasad, N. Dinesh, A. Lee, E. Miltsakaki, L. Robaldo, A. Joshi, and B. Webber. 2008. The Penn discourse treebank. In The Sixth International Conference on Language Resources and Evaluation (LREC’08). find
  21. Sarah E. Schwarm and Mari Ostendorf. 2005. Reading level assessment using support vector machines and statistical language models. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics. find our card
  22. Luo Si and Jamie Callan. 2001. A statistical model for scientific readability. In Proceedings of the Tenth International Conference on Information and Knowledge Management. find
  23. Radu Soricut and Daniel Marcu. 2006. Discourse generation using utility-trained coherence models. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/feng-2010-comparison-features-automatic-readability-assessment/