📄 Exploring Automated Essay Scoring for Nonnative English Speakers

Summary

The paper looks at automated essay scoring (AES) for writers whose first language is not English. Generic scorers handle these writers poorly. Their errors can hide what an essay is about, and some correct local words, such as Indian place names, fail standard spell checks. The author collected more than 900 test essays from undergraduates whose native language is Hindi. Two English professors scored each essay. A feature-based scorer was then trained to reproduce their marks. The features are:

• length-normalised grammar and spelling error measures, with grammar errors sorted into severity buckets
• Flesch–Kincaid readability
• lexical density
• LSA-based topic relevance, computed after grammatical error correction
• entity-grid coherence

A self-correcting lexicon accepts recurring unknown spellings once they appear often enough. Among several Weka learners, Random Forest matched the human scores best, with a correlation of 0.750. The work matters because it adapts AES to a single first-language group. It also shows that error density and lexical complexity carry much of the human judgement.

Key points

• Data: over 900 essays of 150–400 words (mean 247, SD 43) on 7 everyday topics, such as corruption and climate change. All writers were Hindi-native undergraduates.
• Human scoring: two English professors rated each essay on a 1–10 scale, with inter-rater Cohen's kappa of 0.81.
• Tools and setup: LanguageTool detected grammar errors, and Weka handled feature selection (via PCA) and learning. Data was split roughly 60:20:20 into train, validation and test sets.
• Feature correlations with the human score:

• Learner comparison:

The reported correlations are significant at the 0.01 level.

Interesting ideas

• I1. Grammar errors are grouped into severity buckets, such as agreement or tense versus style or capitalisation, with weights based on how harshly human raters penalise each kind, so not every error counts equally.
• I2. An unknown spelling or phrase is promoted into the accepted vocabulary once its total frequency across the corpus passes a threshold, so legitimate local names stop being flagged as errors.
• I3. Grammatical error correction is applied first, and only then is the text compared semantically, because learner multi-word errors confuse topic detection.
• I4. Error counts are normalised per 100 words, and repeated errors receive a damped penalty. A separate "unique errors" measure then separates many different errors from one hard word repeated.
• I5. An essay counts as on-topic if it is similar enough to at least one reference essay, rather than to an averaged prototype. This also serves as a guard against off-topic or padded submissions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): self-induced repair rules and evaluation could carry a severity class, so a broken agreement edge counts for more than a punctuation slip when ranking rules and measuring failures.
• I2 (tokens): the en lexicon could queue repeated out-of-vocabulary forms, such as foreign names, for automatic admission as proper nouns, instead of tagging them as misspellings on every occurrence.
• I3 (tree): a light normalisation pass before UD parsing of learner or noisy text could keep the parser from building strange trees around fixable mistakes, while logging each edit for explainability.
• I4 (learn): parser and tagger error reports could normalise by length and split type-level from token-level failures, so one recurring mistake does not dominate the picture.
• I5 (answer): when grading open answers in school-course tests, a student answer could be accepted as relevant if it matches any one of several reference answers. This tolerates different valid phrasings.