📄 The Ups and Downs of Preposition Error Detection in ESL Writing

Summary

Prepositions are among the most frequent trouble spots for non-native English writers. Their usage is hard to pin down: in a cloze test on 200 Encarta sentences, two native speakers agreed with each other, and with the original text, only about 75% of the time. The authors build a detector for wrong-choice and extraneous preposition errors. Its core is a Maximum Entropy classifier that models correct usage of 34 common prepositions. It is trained on 7 million contexts from the MetaMetrics Lexile corpus and wrapped in filters and thresholds tuned for precision. On student essays it reaches about 84% precision and roughly 19% recall, competitive with leading systems, and it is deployed in Criterion.

The second contribution concerns evaluation. The authors argue that judging such systems against a single annotator can distort the reported performance. As a cheaper alternative to exhaustive multi-rater annotation, they propose a sampling scheme whose precision and recall estimates come close to those from full annotation. Finally, they analyse which preposition errors learners make most often. The evaluation argument extends to other usage errors, such as determiners and collocations.

Key points

• Model features: 25 features drawn from words and POS tags within ±2 tokens of the preposition. Phrase heads are added: the preceding verb (PV), the preceding noun (PN) and the following head noun (FH). The text is POS-tagged and heuristically chunked instead of parsed, because parsers struggle with learner text.
• Combination features: these encode syntactic patterns as word, tag, or word+tag conjunctions (e.g., N-p-N → "place-line" / "NN-NN"). Comlex verb–preposition preferences did not help, and Google Terabyte N-gram queries gave only marginal gains.
• Precision-oriented pipeline: contexts with misspellings are skipped, and rule filters handle antonym pairs (with/without) and benefactive "for" with a human noun. Two rules target extraneous errors ("some of people", repeated prepositions). Probability thresholds are set on a development corpus. Extraneous errors make up to 18% of preposition errors, and the rules catch about a quarter of them.
• Sampling evaluation: sampling gave 79% precision and 14% recall, close to 80% and 12% under the standard exhaustive approach. The sample from the "OK" sub-corpus was twice the size of the sample from the "Error" sub-corpus (~500 vs ~250 contexts). The gold set used in the initial evaluation had 8,269 contexts annotated by two raters.
• Learner error profile: in (21.4%), to (20.8%) and of (16.6%) are the most misused prepositions. The top ten account for 93.8% of errors. Extraneous to/of alone make up 16.8%.

Interesting ideas

• I1. Measure human agreement before building a system. A quick cloze experiment with native speakers gives a realistic ceiling for an ambiguous task. It also warns that single-annotator gold data may be unreliable. This is useful for any task where several answers can be acceptable.
• I2. Learn from clean text, then flag what the model finds unlikely. Train a model of correct usage on well-edited text. Flag a writer's choice only when its probability is low and clearly below the model's top choice. This turns a selection model into an error detector without needing a large annotated learner corpus.
• I3. Optimise for few false alarms in feedback tools. Stack cheap rule filters and two thresholds (the margin over the writer's choice, and an absolute ceiling on the writer's probability). This is a practical pattern wherever false positives mislead users, such as spelling or grammar checkers for other languages.
• I4. Give a linear model hand-built conjunctions. A linear model like ME does not capture feature interactions by itself. Pairing head words with their POS-tag counterparts adds this structure, and the tag versions reduce sparsity.
• I5. Use stratified sampling for evaluation. Annotate samples from the system-flagged and unflagged portions instead of exhaustively labelling everything. Oversample the low-error stratum. This saves rater effort, so several raters become affordable, though the sample must be redone for each system evaluated.