📄 Data-Driven Correction of FunctionWords in Non-Native English

Summary

The paper tackles the automatic detection and repair of wrongly chosen function words in academic English written by non-native authors. It was a submission to the Helping Our Own (HOO) Shared Task. The authors build on an earlier preposition-prediction method by Elghafari, Meurers and Wunsch (2010). For each function word, that method fills the slot with every candidate from a fixed set and picks the filler whose surrounding n-gram is most frequent. The search backs off from 7-grams down to 3-grams.

The authors made three changes. They extended the method to conjunctions, determiners and quantifiers. They replaced web search counts with the ACL Anthology Reference Corpus (ARC), which matches the genre. They also let the context be partly abstracted using TreeTagger POS tags and lemmas. When no evidence is found, the system now keeps the writer's original word.

Their best run caught between a third and two thirds of the targeted substitution errors and proposed a suitable fix for roughly half of them. However, it also changed many correct words. The work shows that a simple count-based method can find errors, and that it needs a strong preference for leaving text unchanged.

Key points

• The task covers single-word substitution errors in four categories. Candidate lists were taken from the HOO training data: 6 conjunctions, 9 determiners, 17 prepositions and 6 quantifiers. These error types are 12% of all errors annotated in that data.
• Run #2 was the best submission by detection score. It used ARC with limited POS/lemma substitutions: POS tags for CD, SYM and LS, and lemmas for comparative adjectives and most verbs. It detected 67% of preposition and determiner errors, 40% of conjunction errors and 33% of quantifier errors.
• Simply keeping every original word already scores 97.2% overall accuracy. Per-category figures range from 94.9% to 98.9%, yet this baseline finds no errors at all.
• The minimum n-gram length controls a strong trade-off:
  • At 3-grams, global accuracy, precision and recall are all near 70%. Detection F-score exceeds 80%, and correction F-score is 44%.
  • Requiring 7-grams pushes precision to about 97%, but recall collapses to 1.5% because of sparsity.
  • Moving from 3-grams to 5-grams raises correction precision from 45% to 60%.
• The system makes roughly ten wrong changes for every true error it flags, which gives a HOO detection precision of 14% in the "no bonus" condition. Some of the errors it finds sit inside other HOO categories, such as phrasal verb or verb form errors, because the annotation scheme lacks finer granularity.

Interesting ideas

• I1. Fill one slot with every member of a closed word class, look up how often each version appears in a reference corpus, and back off to shorter overlapping windows when longer ones are unseen. This is a transparent, count-based way to judge a function word choice.
• I2. Make the context only partly abstract: replace numbers and symbols with POS tags, and verbs and comparatives with lemmas, while leaving other words as they are. This reduces sparsity without discarding the cues that matter, such as "a" versus "an".
• I3. When the evidence runs out, fall back to the original token rather than the most frequent candidate. The authors also suggest adding an explicit bias toward leaving the text unchanged, since most function words are already correct.
• I4. Report results from two angles: accuracy over all words and recall over the actual errors. Also keep in mind that a do-nothing baseline already scores 97%.
• I5. Choosing a function word from corpus counts of a candidate cohort, with longer contexts giving more precise choices, also suits text production.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Self-induced repair rules for prepositions and determiners could be checked against such corpus support. Each accepted fix would then come with an explainable count-based justification.
• I2 (tree): Preposition attachment decisions or perceptron parser features could use contexts abstracted in this mixed way, so that rare surface forms still borrow evidence from frequent ones.
• I3 (tokens): Any normalisation or spelling-repair step in crate en should keep the input unless the evidence for a change clearly wins. This avoids the paper's problem of about ten wrong changes per true error.
• I4 (learn): Evaluations of repair rules should always include unchanged correct text as a negative control, and should report the false-change rate next to the errors caught.
• I5 (back): When generating text from the meaning graph, prepositions, determiners and quantifiers could be chosen by scoring candidate fillers against a genre-matched corpus.