📄 HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task

Summary

Preposition and determiner mistakes are among the most frequent problems for non-native writers of English. Earlier work on correcting them was hard to compare because each group used its own data and task definition. HOO 2012 addressed this with a shared task offering common training data, a common evaluation framework, and unseen test texts. The data came from the CLC FCE Dataset, exam scripts from the Cambridge ESOL First Certificate in English. The organisers kept the original learner texts with all their other errors intact and supplied gold-standard standoff annotations for six preposition and determiner error types. Systems were scored on three increasingly strict subtasks: detection, recognition and correction. Fourteen teams completed the task. After submission, teams found annotation gaps in the test data, so results were reported against both the original and a corrected gold standard. The exercise produced a reusable benchmark, public training data and scoring tools, and practical lessons on gold-standard quality and evaluation strictness.

Key points

• Data and task scope:
  • Training data: 1000 FCE files, 374,680 words, 8,432 annotated errors.
  • Test data: 100 previously unseen files from CUP, 18,013 words.
  • Error types: six tags, RT, MT, UT (preposition replacement, missing, unwanted) and RD, MD, UD (the same for determiners).
• Scoring: systems were scored on detection, recognition and correction. Each subtask was computed for prepositions, determiners, and both combined, giving up to nine scores per run.
• Participation:
  • 26 teams registered and 15 requested the test data; 14 finished.
  • Teams had 10 weeks for development and one week to submit test results.
  • Only ET and CU used privately held data.
  • Author L1 was available in training but withheld in the test data.
• Gold-standard revision:
  • Teams filed 357 revision requests, of which 205 led to changes.
  • Test error counts rose from 453 (Test A) to 473 (Test B).
  • On determiners, NU's detection F-score rose from 40.24 before revisions to 45.8 after.
• Availability: training data and evaluation tools are public, but the test set is not, because of the CUP contract.

Interesting ideas

• I1. Realistic noisy data: errors of other types were left in the training and test texts rather than cleaned out. Benchmarks for any correction or tagging task are more realistic when the target phenomenon sits among natural noise.
• I2. Graded evaluation levels: detection, then recognition (exact extent and type), then correction. This shows where a system fails: spotting a problem, localising it, or fixing it.
• I3. Post-hoc, team-driven gold-standard repair: participants could flag annotation gaps after submitting, and their already-submitted outputs were rescored. Reporting both original and revised scores keeps this transparent, since teams could otherwise bias the gold standard towards their own outputs. This approach could suit any annotation project where system outputs reveal missed errors.
• I4. Offset strictness as a design choice: character-offset matching penalises tiny boundary slips, which inflates the gap between detection and recognition scores. Token-level matching is a fairer alternative for many evaluations.
• I5. A fixed standard test split: the organisers plan to freeze a standard test set, similar to WSJ Section 23. This makes results comparable across years and newcomers, which would help any emerging-language benchmark, including Ukrainian ones.