📄 SemEval 2016 Task 11: Complex Word Identification

Summary

This paper reports on the first shared task on Complex Word Identification (CWI) at SemEval 2016. CWI decides which words in a sentence a simplifier should replace. The organizers argue that earlier resources were unreliable, such as the CW corpus built automatically from Simple English Wikipedia edits. To address this, they ran a user study in which 400 non-native English speakers marked the content words they did not understand in 9,200 sentences drawn from LexMTurk, the CW corpus and aligned Simple Wikipedia. A small portion was labelled by 20 annotators each and used for training. The much larger remainder was labelled by a single annotator per sentence and used for testing. This setup mimics predicting an individual reader's needs from a sample of the population. In total, 21 teams submitted 42 systems, which were compared against nine baselines and ranked by F-score and G-score. Decision Trees, Random Forests and ensembles did best, while neural and embedding-based models lagged behind. Plain corpus frequency remained the most dependable signal of complexity. The work matters because it supplies a reader-grounded benchmark and shows that editor-derived simplification data poorly reflects what non-native readers actually struggle with.

Key points

• Dataset scale: The training sets cover 2,237 instances, each judged by 20 people. The test set contains 88,221 single-annotator instances. Training data came in two forms: a "joint" set with one binary label per word (complex if any annotator flagged it) and a "decomposed" set with all 20 individual labels.
• Low agreement: Krippendorff's Alpha on the multiply-annotated subsets averaged 0.244 (std 0.1). Cohen's Kappa was avoided because the heavy skew toward simple words inflates chance agreement.
• Editor judgments vs. readers: Words simplified by Wikipedia editors were rarely confirmed as complex by the annotators. Only about a third or fewer of the target words in CW, LexMTurk, the SemEval 2012 LS data and LSeval were flagged by anyone. Even 27.3% of the "already simple" Simple Wikipedia sentences contained a word someone found complex.
• Properties of complex words: These words had lower probability under a Simple Wikipedia trigram LM, fewer WordNet senses, synonyms, hypernyms and hyponyms, and were on average shorter. All of these differences were significant.
• Results: The best systems reached roughly 0.77 G-score. The top F-score came from PLUJAGH-SEWDFF, which simply learns a Wikipedia frequency threshold. Neural and embedding systems placed no better than 20th in G-score.

Interesting ideas

• I1. Population-to-individual evaluation split: The organizers train on densely annotated items and test on items labelled by single, different people. This directly measures how well group-level difficulty predicts one reader's needs, and the design could carry over to personalization tasks such as readability for specific learner groups.
• I2. Releasing both aggregated and per-annotator labels: Publishing a "joint" and a "decomposed" version lets participants decide how to handle disagreement themselves. This is valuable whenever labels are inherently subjective.
• I3. Auditing existing resources against target users: The authors checked whether widely used LS/CWI datasets matched real reader judgments, and the mismatch exposed hidden assumptions behind editor-made data. A similar audit is worth doing before reusing any proxy-labelled corpus for a new audience.
• I4. Strong frequency baselines and raw counts: A single learned frequency threshold beat elaborate models on F-score, and raw counts outperformed LM probabilities. This argues for always including a well-tuned frequency baseline and paying attention to corpus choice.
• I5. Choosing the agreement metric to fit label imbalance: The organizers preferred Krippendorff's Alpha over Kappa because of the heavily skewed classes. This is a useful reminder whenever annotating rare phenomena such as errors or complex terms.