Elamur Research⌕
☰
home / papers / 2008 / Revisiting Readability: A Unified Framework for Predicting Text Quality

Revisiting Readability: A Unified Framework for Predicting Text Quality

Summary

Pitler and Nenkova investigate which linguistic properties predict how well written human readers judge a text to be. All the texts come from the same adult audience, so grade-level cues offer little help. They took 30 Wall Street Journal articles with annotations from the Penn Discourse Treebank (PDTB) as well as the Penn Treebank and had college students rate them. They then tested many features against those ratings: classic surface metrics, vocabulary language-model likelihoods, syntactic measures, cohesion counts, Barzilay–Lapata entity grid transitions, and gold-standard PDTB discourse relations. They tried two setups. The first is correlation and linear regression on absolute scores. The second is pairwise ranking with a linear SVM. Traditional surface measures such as word and sentence length mostly fail, and article length is the only one with a significant correlation. Discourse relations and vocabulary are the strongest individual signals. In the ranking task, combining all feature classes reaches about 89% accuracy. The paper matters because it is the first empirical demonstration that discourse structure is tied to perceived text quality. It also shows that which readability predictors matter depends on the task formulation.

Key points

  • Data and ratings:
    • The dataset is 30 randomly chosen WSJ articles; 29 of them have Penn Treebank parses.
    • Each article was rated by at least three college students on a 1–5 scale.
    • Raters answered four questions, but their answers were nearly always identical, so only the "how well-written" rating was kept.
    • Article scores range from 1.5 to 4.33 (mean 3.2008, SD 0.7242).
  • Surface baselines: Characters per word, words per sentence and maximum sentence length are not significant predictors. Article length is significant (r = −0.37), with longer articles rated lower.
  • Regression:
    • Three predictors together (discourse, vocabulary and length) explain about half the variance (R² = 0.5029).
    • The best eight-feature model reaches R² = 0.776. It adds parse tree depth, two cohesion features and several entity grid transitions.
  • Ranking setup:
    • Article pairs whose ratings differ by at least 0.5 give 243 examples.
    • Each example is represented by the differences between the two articles' features.
    • The classifier is WEKA SMO (a linear SVM), evaluated with 10-fold cross-validation.
  • Ranking results:
    • All features: 88.88% accuracy, against a 50.21% majority baseline.
    • Single feature classes: entity grid alone about 79.4%, discourse alone 77.36%, syntax alone 74.07%.
    • The best single feature is the log likelihood of discourse relations (77.77%).
    • Ablation: dropping syntax hurts most (82.71%), while dropping cohesion slightly improves accuracy (89.30%).

Interesting ideas

  • I1. Turn scores into pairwise comparisons. Converting a few absolute quality ratings into pairwise judgments multiplies the number of training examples. It also fits applications that only need relative quality, which is handy for small annotated collections in any language.
  • I2. Use feature differences and a margin filter. Representing a pair by the difference of its feature vectors lets a plain linear classifier do ranking. Skipping pairs with near-equal scores (under 0.5 apart) removes noisy comparisons.
  • I3. Score the discourse sequence probabilistically. Treating a text's sequence of discourse relations as something with a likelihood turns discourse structure into a single strong feature. The same approach could apply to other structured annotations, such as dependency relation sequences.
  • I4. Evaluate in both setups. Testing features in both regression and ranking reveals that their usefulness shifts with the task. Discourse features were the stable ones. This suggests feature studies should not rely on a single evaluation framing.
  • I5. Report class ablations next to single-class results. Doing both exposes interactions between feature groups. For example, entity grid features are weak individually but strong together, and cohesion features can actually hurt. This guards against drawing conclusions from isolated correlations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Discourse analysis · Treebanks

Cited by

154 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 140 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{pitler2008revisiting,
  title = {{Revisiting Readability: A Unified Framework for Predicting Text Quality}},
  author = {Pitler, Emily and Nenkova, Ani},
  booktitle = {Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
  url = {https://aclanthology.org/D08-1020/}
}

References

34 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Attali and J. Burstein. 2006. Automated essay scoring with e-rater v.2. The Journal of Technology, Learning and Assessment, 4(3). find
  2. A. Bailin and A. Grafstein. 2001. The linguistic assumptions underlying readability formulae a critique. Language and Communication, 21(3):285–301. find
  3. R. Barzilay and M. Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1–34. find our card
  4. L. Carlson, D. Marcu, and M. E. Okurowski. 2001. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Proceedings of the Second SIGdial Workshop, pages 1–10. find our card
  5. M. Coleman and TL Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283–284. find
  6. K. Collins-Thompson and J. Callan. 2004. A language modeling approach to predicting reading difficulty. In Proceedings of HLT/NAACL’04. find
  7. Noemie Elhadad and Komal Sutaria. 2007. Mining a lexicon of technical terms and lay equivalents. In Biological, translational, and clinical language processing, pages 49–56, Prague, Czech Republic. Association for Computational Linguistics. find
  8. M. Elsner and E. Charniak. 2008. Coreference-inspired coherence modeling. In Proceedings of ACL-HLT’08, (short paper). find our card
  9. E. Gibson. 1998. Linguistic complexity: locality of syntactic dependencies. Cognition, 68:1–76. find
  10. P. Gordon, B. Grosz, and L. Gilliom. 1993. Pronouns, names, and the centering of attention in discourse. Cognitive Science, 17:311–347. find
  11. B. Grosz, A. Joshi, and S. Weinstein. 1995. Centering: a framework for modelling the local coherence of discourse. Computational Linguistics, 21(2):203–226. find
  12. Robert Gunning. 1952. The technique of clear writing. McGraw-Hill; Fouth Printing edition. find
  13. Michael A.K. Halliday and Ruqaiya Hasan. 1976. Cohesion in English. Longman Group Ltd, London, U.K. find
  14. M. Heilman, K. Collins-Thompson, J. Callan, and M. Eskenazi. 2007. Combining Lexical and Grammatical Features to Improve Readability Measures for First and Second Language Texts. Proceedings of NAACL HLT, pages 460–467. find
  15. D. Higgins, J. Burstein, D. Marcu, and C. Gentile. 2004. Evaluating multiple aspects of coherence in student essays. In Proceedings of HLT/NAACL’04. find
  16. N. Karamanis, M. Poesio, C. Mellish, and J. Oberlander. (to appear). Evaluating centering for information ordering using corpora. Computational Linguistics. find
  17. JP Kincaid. 1975. Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. find
  18. A. Knott, J. Oberlander, M. ODonnell, and C. Mellish. 2001. Beyond elaboration: The interaction of relations and focus in coherent text. Text representation: linguistic and psycholinguistic aspects, pages 181–196. find
  19. E. Krahmer and M. Theune. 2002. Efficient context-sensitive generation of referring expressions. In K. van Deemter and R. Kibble, editors, Information Sharing: Reference and Presupposition in Language Generation and Interpretation, pages 223–264. CSLI Publications. find
  20. M. Lapata. 2006. Automatic evaluation of information ordering: Kendalls tau. Computational Linguistics, 32(4):471–484. find
  21. W. Mann and S. Thompson. 1988. Rhetorical structure theory: Towards a functional theory of text organization. Text, 8. find
  22. M.P. Marcus, B. Santorini, and M.A. Marcinkiewicz. 1994. Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  23. G.H. McLaughlin. 1969. SMOG grading: A new readability formula. Journal of Reading, 12(8):639–646. find
  24. E. Miltsakaki and K. Kukich. 2000. The role of centering theory’s rough-shift in the teaching and evaluation of writing skills. In Proceedings of ACL’00, pages 408– 415. find
  25. A. Nenkova and K. McKeown. 2003. References to named entities: a corpus study. In Proceedings of HLT/NAACL 2003 (short paper). find
  26. E. Pitler, M. Raghupathy, H. Mehta, A. Nenkova, A. Lee, and A. Joshi. 2008. Easily identifiable discourse relations. In Coling 2008: Companion volume: Posters and Demonstrations, pages 85–88, Manchester, UK, August. find our card
  27. M. Poesio and R. Vieira. 1998. A corpus-based investigation of definite description use. Computational Linguistics, 24(2):183–216. find our card
  28. R. Prasad, N. Dinesh, A. Lee, E. Miltsakaki, L. Robaldo, A. Joshi, and B. Webber. 2008. The penn discourse treebank 2.0. In Proceedings of LREC’08. find our card
  29. KA Schriver. 1989. Evaluating text quality: the continuum from text-focused toreader-focused methods. Professional Communication, IEEE Transactions on, 32(4):238–255. find
  30. S. Schwarm and M. Ostendorf. 2005. Reading level assessment using support vector machines and statistical language models. In Proceedings of ACL’05, pages 523–530. find our card
  31. L. Si and J. Callan. 2001. A statistical model for scientific readability. Proceedings of the tenth international conference on Information and knowledge management, pages 574–576. find
  32. A. Siddharthan. 2003. Syntactic simplification and Text Cohesion. Ph.D. thesis, University of Cambridge, UK. find
  33. V. Spandel. 2004. Creating writers through 6-trait writing assessment and instruction. Allyn & Bacon. find
  34. F. Wolf and E. Gibson. 2005. Representing discourse coherence: A corpus-based study. Computational Linguistics, 31(2):249–288. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pitler-2008-revisiting-readability-unified-framework-predicting/