📄 Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition

Summary

The paper examines a known weakness of named entity recognition: systems score well on standard benchmarks mainly because those benchmarks repeat frequent, familiar names, and they break down on rare or newly appearing entities in noisy social-media text. The WNUT 2017 shared task set out to measure how well systems generalise to such unseen names. The organisers trained on an existing Twitter NER corpus. They built new development and test sets from YouTube, Twitter, Reddit and StackExchange comments, and removed every text containing a surface form that was already labelled as an entity in the training set. Crowd workers annotated six entity types. Systems were scored with standard entity-level F1 and with a new surface-form F1 that counts each distinct (name, type) pair only once. Seven teams took part. The best result, from UH-RiTUAL, was roughly 42 entity F1, far below typical newswire numbers. This confirms that emerging and long-tail entities remain hard. The work matters because it provides a reusable benchmark and metric that reward generalisation rather than memorisation.

Key points

• Training data: the Ritter et al. (2011) Twitter corpus, with 1,000 tweets and 65,124 tokens.
• Development set: YouTube comments only, with 1,008 documents, 15,734 tokens and 835 entities.
• Test set: Twitter, Reddit and StackExchange, with 1,287 documents, 23,394 tokens and 1,040 entities.
• Entity classes: person, location (covering GPE and facility), corporation, product, creative-work and group.
• Annotation: done through CrowdFlower using the GATE crowdsourcing plugin, with three annotators assigned to each document.
• Results: seven systems were scored on F1 (entity) and F1 (surface). The top system was UH-RiTUAL with 41.86 / 40.24, followed by SpinningBytes with 40.78 / 39.33. The weakest was Drexel-CCI with 26.30 / 25.26.
• Error analysis:
  • Systems often cut off names that contain common words, such as tagging only "Andrew" in "Andrew Little".
  • They mislabelled non-English names, for example the Kashmiri village "Mahazgund".
  • Corporation and creative-work were the hardest classes. Typical problems were confusion between corporation, group and product, and long titles built from ordinary words.

Interesting ideas

• I1. Novelty by construction: collect all entity strings from the training set into a blacklist and drop any evaluation text that contains one of them. This cheaply guarantees a truly unseen test set. It could be applied to any Ukrainian NER or gazetteer-heavy task where memorisation inflates scores.
• I2. Surface-form F1: score unique (surface form, type) tuples rather than individual mentions, so a frequent name such as a capital city counts only once. This exposes whether a model handles diverse names or just the head of the distribution. It is a useful extra metric wherever entity frequencies follow a Zipfian distribution.
• I3. Deliberate domain drift between splits: use different source mixtures for development and test data, such as YouTube for development and Reddit, Twitter and StackExchange for test. This creates an intentionally hostile setting that tests robustness instead of overfitting to one platform. It is a good design for stress-testing models before real-world deployment.
• I4. Pre-filtering for entity-bearing text: keep only sentences in which two independent POS taggers both mark at least one proper noun (NNP). This raises the density of entities before paying for annotation, and the same trick could cut crowdsourcing costs for new corpora.
• I5. Sampling around specific events and regions: draw data from periods of recent disasters and from city- or country-level forums to surface novel place names. This is worth borrowing for geography-oriented datasets that need rare local toponyms.