📄 Application of a Corpus to Identify Gaps between English Learners and Native Speakers

Summary

Computer-assisted teaching tools aimed at people studying English as a non-native language need to know where learners fall short of native speakers. Existing comparative learner corpora covered only writing and speaking. The authors extend an earlier learner corpus (Kotani et al. 2011), which has Japanese learners at three TOEIC-based levels with 30 people each, by adding 30 native speakers who performed the same tasks. The tasks are sentence-by-sentence silent reading, picture-description writing, reading aloud, and listening to news clips. The combined resource records accuracy (comprehension or correctness rates) and fluency (words per minute and self-rated difficulty on a five-point scale) for all four skills, across four proficiency tiers. Every measure ordered the groups as expected, from beginner up to native, which supports the data's validity. As an application, they fit a regression model that predicts, for each sentence, how much slower advanced learners read it than natives, using only length and syllable features. The goal is to help teachers judge whether authentic web materials suit a given class.

Key points

• The native-speaker group has 30 university students from the Tokyo area (16 male, 14 female, mean age 22.5). It is treated as a fourth tier above the beginner, intermediate, and advanced learner groups.
• Reading, reading aloud, and listening each use 80 sentences drawn from four news articles or audio clips. Reading and listening add five multiple-choice questions per text, and every task includes per-sentence Likert difficulty or confidence ratings.
• Comprehension rose from beginner to native, from 47.3% to 76.5% for reading and from 43.3% to 75.7% for listening. Silent-reading speed rose from about 87 to 206 WPM, writing from about 9 to 17 WPM, and reading aloud from about 66 to 120 WPM.
• The difficulty model is a multiple regression on sentence length in words and mean syllables per word. Its target is the per-sentence silent-reading gap between natives and advanced learners. The fit gave F(2,77)=17.42 with adjusted R=0.54, explaining roughly 31% of the variance.
• Leave-one-out cross-validation over the 80 sentences gave a Spearman correlation of 0.48 (p<0.01). Most absolute prediction errors fell in the 0–25 WPM range.

Interesting ideas

• I1. The model defines sentence difficulty as the measured performance gap between a weaker and a stronger group, not as an abstract readability score, which grounds the label in observed behaviour.
• I2. The authors check the corpus by confirming that every measure orders the groups monotonically from beginner to native, a cheap sanity test before trusting a new dataset.
• I3. Each item records both objective correctness and the subject's own difficulty or confidence rating, so accuracy and perceived effort can be compared directly.
• I4. Two shallow surface features predict a meaningful share of reading-speed gaps, a transparent baseline that is easy to explain.
• I5. With only 80 items, the authors validate using leave-one-out cross-validation scored by rank correlation, which suits tiny data where held-out splits would be unstable.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Per-sentence difficulty could be scored as the gap in accuracy between a weak witness parser and a strong one, which would flag sentences worth sending to the LLM teacher tiers.
• I2 (learn): Evaluation sets could require that scores rise across known-ordered model checkpoints or data tiers, and treat any violation as a negative-control alarm.
• I3 (answer): Student readers could output a self-confidence score next to each test answer, which would allow calibration of when Mova is sure versus guessing.
• I4 (back): During generation, sentence length and syllable counts could serve as explainable controls when producing text for a target reader level.
• I5 (answer): Small hand-built QA sets on fables or maths word problems could be evaluated with leave-one-out and Spearman correlation rather than a single fragile split.