📄 Exploring the Influence of Spelling Errors on Lexical Variation Measures

Summary

Lexical richness scores such as Type-Token Ratio (TTR) and Yule's K are commonly used to analyse and grade learner English. Both depend on how many distinct word types a text contains, and misspellings inflate that count. A word like because spelt three wrong ways would count as four types. The authors test whether this makes the measures unreliable. They use essays by Japanese-L1 learners at three levels (middle school, high school, college), compute TTR and K on the raw text, and compare them with versions where the spelling errors were corrected by hand. They try two correction policies built on a 13-category error taxonomy.

TTR shifts by at least 16% in every group. K changes by no more than 1%. A short derivation explains this: K is driven by squared frequencies of common words, which are rarely misspelt. The authors conclude that K is the safer choice for measuring vocabulary in noisy learner text.

Key points

• Corpus: 1,073 essays, 82,659 words and 2,534 spelling errors across the three groups. Mean edit distance of SP-type errors is 1.38. Part of the data is public as the Konan-JIEM (KJ) learner corpus.
• Taxonomy: 13 error types, including SP, PC, OC, GC, NM, RE and romanized Japanese (RO/SR).
  • The "all" policy corrects every type.
  • The "selected" policy corrects SP, PC, OC, GC and NM, keeps real-word errors (RE) unchanged, and drops the remaining types from both counts.
• Preprocessing: Stanford Parser 3.5.0 splits sentences and tokens. Text is lowercased, and tokens with no Latin letters are removed. Merged-word and split-word errors are excluded.
• Results: TTR differs by at least about 16% between original and corrected text. K differs by at most 1%.
• Misspelt share r: This is the fraction of a word's occurrences that are misspelt.
  • For words with frequency 10–99, r has mean 0.06 (SD 0.10).
  • For words with frequency ≥100, r has mean 0.01 (SD 0.024).

Interesting ideas

• I1. Before/after manual correction as a controlled experiment. Compute a metric on the original text and again on a hand-corrected copy, so the gap isolates the effect of the noise itself. This shows directly how much a statistic depends on surface errors.
• I2. Each error class gets one of three treatments: fix, keep, or exclude. The choice depends on whether the error reflects what the writer actually knows. A real-word mix-up like their/there may show genuine ignorance, so it is kept, while invented forms are removed from the counts.
• I3. Analytic sensitivity check. If a fraction r of a word's occurrences is misspelt, that word's squared-frequency contribution scales by (1−r)²+r², which is close to 1−2r when r is small. A quick derivation like this predicts which statistics will be robust to noise before running experiments.
• I4. Measuring r empirically per frequency band. Splitting words into frequency bands showed that frequent words are almost never misspelt. This supports any assumption that each word has one dominant spelling.
• I5. Choose the metric that survives noise. Rather than only cleaning data, pick a statistic that is inherently stable under the expected noise, as K is here.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Our evaluations could include noisy-vs-clean paired runs, checking how much parser LAS or frame extraction drops on misspelt inputs.
• I2 (tokens): The normalizer could tag each suspect token with a treatment class, so later stages know whether to re-lemmatize it, trust it, or skip it.
• I3 (world): When compiling frequency-based knowledge (measures, verb-class statistics), prefer frequency-weighted aggregates dominated by common items, and document their noise sensitivity formally.
• I4 (tokens): Banded misspelling rates could set thresholds for the spelling-variant lexicon, mapping rare variants to a dominant form only when the canonical form clearly dominates.
• I5 (answer): When scoring student or school-test answers for vocabulary or style, use K-like measures so that misspellings do not distort the assessment.