📄 Measuring Non-native Speakers' Proficiency of English by Using a Test with Automatically-Generated Fill-in-the-Blank Questions

Summary

The paper addresses the high cost of writing fill-in-the-blank (FBQ) English proficiency tests. Good wrong options, which mislead only weaker learners, normally require skilled human writers. The authors build an automatic pipeline that works from a seed sentence taken from business and travel conversation corpora or from Web pages.

The pipeline works in four steps:

• It blanks the verb in the seed sentence.
• It collects same-meaning candidates for the blank from a thesaurus.
• It searches the Web and keeps only candidates whose restored sentence fragment gets no hits at all.
• It builds a four-option item from the correct word and three surviving distracters.

Test-takers' answers are then scored with Item Response Theory (IRT). In the experiments, the estimated proficiency of non-native speakers correlated strongly with their TOEIC scores, while native speakers solved the test almost perfectly. The test could also be shortened by using IRT item information to discard items that tell test-takers apart poorly. This matters because it enables cheap, on-demand testing that can draw on current texts. The authors implemented a mobile-phone version that runs adaptive testing.

Key points

• Each item has one correct answer and three distracters. The blank position is chosen by "testing knowledge," which in the experiments meant the main verb, following TOEIC-style questions.
• Distracter candidates come from an in-house English thesaurus whose hierarchy follows the Japanese thesaurus Ruigo-Shin-Jiten. Example: for "keep," the candidates "clear" and "share" were rejected, while "guarantee," "promise," and "reserve" were kept.
• Web verification used Google (about four billion pages as of February 2004) under the strictest rule: a candidate survives only if it gets zero hits. Instead of the full sentence, the query is a word window around the blank, running from a content word or sentence start to a content word or sentence end.
• IRT models the probability of a correct answer as an S-shaped logistic function of ability. Item information is used to remove items that do not discriminate between test-takers.
• Questions were also generated from Japan Times news articles and from over 100 Project Gutenberg e-books. A grammar variant keeps the word stem and varies only its inflected form. Listening skills are out of scope.

Interesting ideas

• I1. The method separates generation from verification: a lexical resource proposes plausible same-category substitutes, and an external evidence source filters out any substitute that still yields an acceptable sentence, which gives cleaner contrastive examples than random corruption.
• I2. Attestation is checked on a short local window bounded by content words rather than on the whole sentence, because whole-sentence lookups almost never match while local fragments capture the key collocation.
• I3. The generated test is validated against groups of known ability: natives should score near the ceiling, and non-native scores should track an external benchmark (TOEIC).
• I4. IRT item information is used to keep only discriminative items, so a much smaller test measures nearly as well as the full one.
• I5. In the grammar variant, the stem stays fixed and only the inflected form changes, so the item tests grammar rather than word meaning.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Build negative-control minimal pairs by swapping a verb for its VerbNet/WordNet neighbours and keeping only unattested swaps, then check that the pipeline flags them as anomalous.
• I2 (frames): Score verb–argument plausibility for semantic-role and frame decisions by looking up the local predicate–argument window in a large corpus, keeping the evidence as an explainable trace.
• I3 (answer): Validate auto-generated school-style test items by checking that strong reference readers ace them and that weaker reader configurations score lower in the expected order.
• I4 (learn): Fit IRT over many parser versions or witness parsers and prune evaluation items that every system passes or fails, yielding a compact, sensitive regression suite.
• I5 (tokens): Use the morphology generator to produce inflection-variant distracters and test whether the tagger and lemmatizer, plus the downstream tree, reject the wrong surface forms.