📄 WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets

Summary

This paper describes the WNUT-2020 shared task on deciding whether an English COVID-19 Tweet is INFORMATIVE or UNINFORMATIVE. A Tweet is informative when it reports concrete facts such as suspected, confirmed, recovered or fatal cases, test counts, or the location and travel history of cases. Such Tweets are useful for pandemic monitoring, because official sources update slowly and most Tweets carry no usable information.

The organisers streamed Tweets from March to June 2020 and filtered them down to about 23M. From these they built a 10K gold corpus in two annotation rounds with three annotators each. In the second round, a fastText model trained on the first round helped choose items across a range of difficulty.

Of 121 registered teams, 55 submitted final results. The best systems reached about 0.91 F1, well above the fastText baseline. The strongest approach was transformer models with domain-adapted pre-training, mainly CT-BERT, often combined in ensembles. The dataset is a public benchmark for filtering noisy social media text.

Key points

• Corpus and split: 10,000 labelled Tweets (4,719 INFORMATIVE, 5,281 UNINFORMATIVE), divided into 7K train, 1K validation and 2K test.
• Collection and filtering:
  • Tweets were gathered with 10 COVID keywords.
  • Tweets under 10 words were dropped, as were Tweets from accounts with fewer than 500 followers.
  • Retweets were removed, and near-duplicates were removed after normalisation.
  • Candidate Tweets were picked by trigger strings such as "confirm" and "death", then deduplicated at token cosine ≥ 0.7.
• Annotator agreement: Fleiss' Kappa was 0.797 in phase 1 (2K Tweets) and 0.818 in phase 2 (8K Tweets). Disagreements were settled through discussion.
• Evaluation setup:
  • Systems were ranked by F1 on the INFORMATIVE class.
  • The 2K test items were hidden inside a 12K raw set.
  • Each team could make at most 2 submissions.
• Results:
  • The fastText baseline scored F1 0.7503. 48 teams beat it, 39 passed 0.80 and 10 passed 0.90.
  • NutCracker and NLP North tied for first at 0.9096.
  • All of the top 6 used CT-BERT, and 61% of teams used ensembles.

Interesting ideas

• I1. The organisers used a weak baseline's predicted probability to sample new items in fixed confidence bands (3K low, 2K middle, 3K high), so the labelled data covers easy and hard cases on both labels instead of mostly obvious ones.
• I2. Items where annotators disagreed were tagged "not-easy" and spread evenly across train, validation and test, so every split has a similar share of hard examples.
• I3. The true test items were buried in a much larger unlabelled pool, so participants could not tune by hand on the scored portion.
• I4. The labelling guideline separates exact or directly derivable counts, which count as informative, from bare percentages or rates, which do not.
• I5. The guideline was refined after the first round, using discussion of disagreements, before the larger second round was labelled.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): When choosing sentences to send to LLM teacher tiers, Mova could sample across bands of witness-parser agreement, so the teacher budget also covers borderline sentences and not only clear-cut ones.
• I2 (tree): When splitting treebanks or silver data for gparse and the perceptron parser, Mova could spread sentences where witness parsers disagree evenly across splits, keeping LAS comparisons between splits fair.
• I3 (learn): For evaluation with negative controls, Mova could mix gold-checked sentences into larger unscored batches, so self-induced repair rules cannot quietly overfit to the known test items.
• I4 (world): The text-world builder could tag numeric mentions as absolute quantities, derivable quantities or ratios, so holdings and states only take values that can actually be computed.
• I5 (learn): Mova's annotation conventions for silver frames could be tightened the same way: run a small pilot, review where the witnesses or teachers disagree, then scale up.