📄 SemEval-2021 Task 1: Lexical Complexity Prediction

Summary

This shared-task report argues that a binary "complex vs. simple" label for words loses information, because readers differ in background and in how strictly they define difficulty. The organisers therefore replace Complex Word Identification (CWI) with Lexical Complexity Prediction (LCP), where systems output a graded score between 0 and 1. They extended the CompLex corpus into CompLex 2.0 by collecting about ten extra crowd judgments per item on Amazon Mechanical Turk and merging them with the retained CompLex 1.0 labels. Each Likert answer was mapped to a number and averaged. The resulting 10,800 in-context targets come from three genres and include both nouns and two-word expressions. These targets were split into two Sub-tasks: single words, and MWEs. Many teams took part. The best systems correlated well with human scores and clearly beat a frequency baseline, which earlier CWI tasks often failed to do. The work matters because a graded complexity score can feed lexical simplification, readability assessment and other downstream tasks. When needed, the score can still be turned into a binary label with a threshold.

Key points

• CompLex 2.0 has 10,800 instances drawn from Europarl, the World English Bible and the CRAFT biomedical corpus. Targets are nouns plus Noun-Noun and Adjective-Noun bigrams, identified with the Stanford CoreNLP POS tagger.
• The five Likert levels were converted to 0, 0.25, 0.5, 0.75 and 1.0, and the mean of these values became the gold label. CompLex 1.0 had kept only items with at least 4 annotations, averaging about 7 per item; roughly 10 more were added for 2.0.
• 198 teams registered. 54 submitted official test runs for Sub-task 1 (single words) and 37 for Sub-task 2 (MWEs), with 55 teams active across both.
• Top Pearson correlations were 0.7886 for Sub-task 1 and 0.8612 for Sub-task 2, with R2 scores of 0.6172 and 0.7389.
• The spread between rank 1 and rank 10 was tiny: 0.0182 Pearson points for Sub-task 1 and 0.021 for Sub-task 2. Biomedical text and MWEs tended to be rated harder than average.

Interesting ideas

• I1. Average graded judgments instead of collecting yes/no votes. Mapping Likert answers to evenly spaced numbers and averaging them preserves annotator disagreement as a mid-range value rather than forcing an arbitrary decision. This suits any subjective labelling task, such as toxicity, formality or text difficulty for Ukrainian learners.
• I2. Annotate the same word in several contexts. Including up to five sentences per token per genre means a system that ignores context is automatically penalised. This is a simple way to make a dataset measure context sensitivity rather than lexicon lookup.
• I3. Define every scale point in terms of reader experience. The levels were described concretely, for example "never seen before" or "unclear but inferable from the sentence." Tying labels to behaviour like this can make crowdsourced annotations more consistent across languages and annotator pools.
• I4. Treat small leaderboard gaps with caution. The organisers note that the top-10 differences are so small that a different random split might reorder the ranking. Reporting such gaps, or testing robustness across splits, is a good habit for any benchmark or shared task.
• I5. Pick a continuous target that can still be thresholded. Predicting a score allows systems to rank words and still produce binary decisions when a downstream tool needs them. One dataset can therefore serve simplification, readability and feature-engineering uses at once.