📄 Holistic Discourse Coherence Annotation for Noisy Essay Writing

Summary

Automated essay scoring needs coherence measures that work on noisy student writing, which often contains grammatical errors, misspellings and weak organization. The features behind these measures should also match the criteria in human scoring rubrics, so that scores can be explained and turned into feedback. Burstein, Tetreault and Chodorow propose a holistic coherence annotation scheme that annotators without deep linguistic training can apply quickly. Annotators read with the aim of judging the text rather than understanding it, then assign an overall coherence rating on a three-point scale. For modeling, these ratings are reduced to a binary low/high label. Most disagreement occurred at the middle level, where readers differ in the inferences they use to bridge gaps in the text. Even so, the binary label reached adequate reliability. Using these labels, the authors trained classifiers on 1555 essays from five assessment types. The models combine entity-grid, type/token, RST, LSA and e-rater grammar-error features. On every data set, the best combination outperformed baselines at detecting low-coherence essays. This shows that inexpensive annotation can support interpretable coherence scoring.

Key points

• The corpus has 1555 essays in five sub-corpora. EP/NNC and Summary/NNC are non-native college-level writing. EP/GL contains graduate-level essays, the PPE set comes from the Professional Proficiency Exam, and EP/6-12 spans grades 6–12.
• Score Point 2 ("mostly coherent," with one or two points of confusion) was the hardest level to agree on. It was merged with Score Point 3 into a single "high" class for modeling.
• The baselines were Majority Class, full e-rater features and ERGramError features. The best systems' F-scores for the "low" class were 74 (EP/NNC), 57 (Summary/NNC), 63 (EP/GL), 39 (PPE) and 64 (EP/6-12).
• The best feature sets usually combined EntityGrid, ERGramError, Type/Token and RST, with maxLSA added for some sets. Summary/NNC was the exception: its best model used only ERGramError and RST.
• Each feature type maps to a rubric criterion:
  • entity-grid transitions reflect how ideas are organized;
  • type/token reflects repetition of ideas;
  • RST reflects the discourse units that organize and develop the text;
  • grammatical errors reflect technical quality.