📄 Helping Our Own: The HOO 2011 Pilot Shared Task

Summary

Dale and Kilgarriff report on the pilot round of Helping Our Own (HOO), a shared task for automatic tools that help authors improve their writing. It targets English papers from the NLP community written by non-native speakers. The authors chose this domain because participants know the subject matter and the ACL Anthology offers plenty of free in-domain text. Non-native writing was chosen because errors occur more densely, which makes annotation cheaper. The data are extracts from 19 published ACL papers, with one development and one test fragment per paper. Professional copy-editors annotated the fragments with stand-off XML edits, which can list several alternative corrections and can mark edits as optional, linked or uncorrectable. Systems were scored on three levels: detection, recognition of exact spans, and correction. Six teams took part, and the best system came from the University of Illinois, with only modest F-scores. The pilot set up a reusable data format and evaluation framework. It also showed that annotating such a broad range of error types is costly, skilled work.

Key points

• Data size: fragments average about 940 words. The development set has 1264 edits (16–100 per file) and the test set has 1057 edits (18–107 per file). Both average roughly one edit per 15 words.
• Annotation: two professional copy-editors annotated each development fragment. For the test fragments, a single copy-editor did the annotation and an organizer then reviewed it. Error types follow the Cambridge University Press Error Coding System (CLC tagset), with an added CC (Compound Change) tag.
• Participants: six teams (JU, LI, NU, UD, UI, UT) from India, France, Singapore, Germany and the USA. Each could submit up to 10 runs, either as corrected plain text or as XML edit structures.
• Best results: under the "no bonus" condition, team UI scored best, with detection F 0.277, recognition F 0.243 and correction F 0.211. Crediting missed optional edits raised its detection F to about 0.366.
• Error type frequencies: the most common aggregated types in the test data were Article (260), Punctuation (206), Preposition (121), Noun (113) and Verb (108).

Interesting ideas

• I1. Stand-off gold annotation with alternatives. Each gold edit stores character offsets, the original span and a list of acceptable corrections, rather than a single rewritten text. This fits any task where several fixes are legitimate, such as normalization or transliteration.
• I2. Optional edits. A null correction listed first marks an edit as optional, and a "bonus" scoring variant rewards systems for leaving such spots unchanged. This keeps systems from being punished for conservative choices that are still valid, which matters for stylistic or debatable edits.
• I3. Linked edits. Consistency sets (cset) tie together edits that must be applied jointly, such as hyphenation of one term across a document or number agreement between distant words. This is a neat way to encode document-level dependencies in otherwise local annotations.
• I4. Graded evaluation. Scoring is split into detection (lenient, overlapping spans), recognition (exact spans) and correction. This separates "found the problem" from "fixed it correctly" and gives more diagnostic feedback than a single score.
• I5. Recall per error type. Gold edits are grouped by part-of-speech-based error categories, and recall is reported for each category. Teams that target only some phenomena, such as articles or prepositions, can then be compared fairly, even though systems are not required to label their edits.