📄 Automated Content Scoring of Spoken Responses in an Assessment for Teachers of English

Summary

The paper addresses automatic scoring of content correctness in spoken answers on ETLA, a 2012 English test for non-native EFL teachers. Its target is items whose answers are only moderately predictable: short and tied to a classroom situation, but still varied. Using the word sequence produced by the ASR front end of SpeechRater, the authors built 15 content features in three families: flexible string matching (regular expressions, keywords), n-grams of words or POS tags (including BLEU), and string edit distance (including WER-style measures). They developed the features on a small in-house corpus of expert-written and read-aloud responses, kept seven of them, and then tested those on pilot data from 2,308 test takers. On recognizer output, individual features correlated with human holistic scores at roughly 0.29–0.48. Adding the features to a linear regression scoring model raised its agreement with human scores from 0.33 to 0.56. This matters because, for such brief answers, fluency and pronunciation cues carry little signal, so content features become the main basis for scoring.

Key points

• In-house set: 22 items and 1,053 expert-drafted text responses at score levels 1–3. Of these, 789 were used for development, and 264 were recorded (some by two speakers), giving 444 spoken responses for evaluation.
• Pilot set: 14 moderately predictable items, split into asrTrain/asrDev/asrEval/smTrain/smEval with no speaker overlap.
  • Responses average 10.5 words.
  • Human inter-rater correlation is 0.73.
  • ASR word error rate is about 27%, compared with 18% in-house.
• The re_match feature takes three values: 0 for no match, 1 for a partial match, and 2 for an exact match. It was the strongest feature on transcripts in-house (r = 0.789) but lost about a third of its correlation on ASR output.
• Seven features were kept: three from string matching, three from edit distance, and bleu_3. Their correlations were 0.42–0.60 on transcripts and 0.29–0.48 on ASR output. Fluency and pronunciation features stayed below 0.3 on this data.
• Every selected feature correlated slightly better with human analytic content scores than with holistic scores (by 0.01–0.04).

Interesting ideas

• I1. A pattern-match feature with a middle "partial" level gives credit when the correct answer is embedded in extra material, such as repetitions or self-corrections, instead of failing the whole response
• I2. Features were chosen on three criteria together: predictive strength, low redundancy with other features, and fit with what experts consider content. This avoids piling up correlated signals that measure the same thing
• I3. Each feature was measured twice, on clean human transcripts and on noisy ASR output, and the relative drop was reported. This exposes which features are brittle under realistic input
• I4. The authors checked that a feature tracks the intended construct by showing it aligns better with a narrow content score than with a broad holistic score
• I5. POS-tag n-grams served as a cheap proxy for grammatical well-formedness when comparing against reference answers, alongside BLEU and edit distance

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): student-answer checking in school-course tests could distinguish an exact answer, a correct answer surrounded by noise, and no answer, rather than using a binary match.
• I2 (learn): when choosing evaluation metrics or repair-rule triggers, drop near-duplicate signals and keep ones a human would accept as meaningful.
• I3 (tree): compare parser performance on gold-tokenized text against noisy or automatically tokenized text and report the relative degradation per component.
• I4 (learn): this works as a negative-control style check, confirming that a frame or world metric correlates more with its targeted dimension than with an unrelated overall quality judgment.
• I5 (back): generated text from the meaning graph could be compared with references at the POS-sequence level as well as the word level, separating grammatical errors from lexical ones.