📄 Automatic Measuring of English Language Proficiency using MT Evaluation Technology

Summary

The paper asks whether tools built to score machine translation can also grade human learners of English. The authors want a fast, computer-run test that does not require experts to write multiple-choice items. Japanese learners translate Japanese sentences into English. Their output is compared with multiple reference translations using a DP (edit-distance) score, BLEU or NIST. A straight regression line then maps the averaged match score onto TOEIC scores. Before testing, each candidate sentence is kept or dropped based on how well its automatic score tracks the subjects' known TOEIC levels.

Two travel-domain corpora were used: BTEC (510 sentences, 18 subjects) and SLTA1 (330 sentences, 29 subjects). Each sentence had 16 references. With the selected sentences, test-set-level correlation with TOEIC reaches about 0.95. In leave-one-out cross validation, the estimation error is roughly 120 TOEIC points on BTEC and under 100 on SLTA1. The best test length is 30–60 sentences. The work shows that reference-based overlap metrics can serve as cheap, compact proficiency probes.

Key points

• Three metrics are compared: a DP-based edit-distance score, BLEU (n-grams up to 4) and NIST (up to 5). All three behave in a similar way.
• The data are BTEC (phrasebook-style, about 8 words per sentence, TOEIC range 400s–900s) and SLTA1 (23 interpreted hotel dialogues, about 13 words per sentence, TOEIC range 300s–800s).
• Each sentence has 16 references: one from the corpus English side, plus 15 written by 5 bilinguals (3 each).
• About 20% of BTEC sentences and about 10% of SLTA1 sentences correlate negatively with TOEIC. These are mostly very short, common phrases, or sentences with many valid translations that the references fail to cover.
• After selection, 30–60 sentences give a standard error of about 120 TOEIC points (BTEC) or below 100 (SLTA1). The authors propose automatic paraphrasing as a future way to produce references.

Interesting ideas

• I1. Each test item is screened by correlating its automatic score with a trusted external measure, and items that fail to discriminate are discarded, which makes a short test stronger than the full pool.
• I2. A low item correlation can point to two different problems: the item is trivial, or the scoring cannot handle the item because its valid answers are too varied.
• I3. A raw overlap metric is turned into a human-meaningful scale by fitting a regression on calibrated subjects, and the result is checked with leave-one-out error rather than correlation alone.
• I4. Many references per sentence reduce the penalty for valid but unusual phrasings, and the authors plan to generate such references through paraphrasing.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Mova's evaluation suites could keep only those sentences whose parser scores track a known ranking of parser versions, giving small but sensitive regression tests.
• I2 (answer): if automatic grading of a school-test or fable-QA item disagrees with known-good students, the item could be flagged as a likely grader blind spot.
• I3 (learn): silver-data agreement scores could be mapped onto human-checked accuracy and reported with held-out error, so the numbers mean something concrete.
• I4 (back): output of the text → graph → text compressor could be scored against several paraphrase references instead of one, so faithful rewordings are not counted as losses.