📄 #GCDH at WNUT-2020 Task 2: BERT-Based Models for the Detection of Informativeness in English COVID-19 Related Tweets

Summary

This system paper tackles WNUT-2020 Task 2, a binary classification task. Systems had to separate English COVID-19 tweets that report case statistics (confirmed, suspected, recovered, deaths) from tweets that carry no such data. The motivation is to feed real-time trackers, since official agencies publish figures only about once a day.

The team used the organizers' split of 7,000 training and 1,000 validation tweets. They first built two classic baselines over token n-grams: a Naive Bayes classifier and an SVM, plus many hand-made extra features. They then fine-tuned two transformers, uncased BERT-large and COVID-Twitter-BERT (CT-BERT). Both baselines plateaued near 0.80 F1. The transformers gained more than ten points, and the domain-adapted CT-BERT came out on top (0.9231 F1 on validation, 0.9091 on the official test).

The work shows that pretraining on in-domain social media text beats generic pretraining. It also shows that feature engineering added nothing over plain n-grams. The authors argue that informativeness detection is only a first step toward checking claims and their timeliness.

Key points

• Baselines:
  • NBC peaked at 0.8012 F1 using uni- and bigrams without number masking.
  • SVM reached 0.8009 with binary uni-/bigram features and masked numbers.
  • None of the extra SVM features improved results: VADER sentiment, Gunning fog index, CORD-19 medical-term counts, Wang et al. (2015) content features, URL/USER position flags and an opinion-phrase list.
• Transformer setup:
  • Preprocessing used the CT-BERT code. User names and links became placeholder tokens, emoji became ASCII, and 64 tweets that ended up empty were dropped.
  • 24 tweets were shifted to validation so batches stayed exactly 32.
  • Training ran on Colab TPUs with sequence length 96, Adam at 2e-5, and per-layer learning-rate decay of 0.95.
• Validation results: BERT-large (2 epochs) scored 0.8928 F1. CT-BERT-1 (3 epochs) scored 0.9231, with recall 0.9487.
• Official test results: CT-BERT-1 (6,912 tweets) got 0.9091 F1. CT-BERT-2, also trained on validation data (7,936 tweets), got 0.9036: precision rose slightly but F1 did not. The roughly 2,000 test tweets were hidden among about 12,000.
• Error analysis: Most of BERT's 115 errors were uninformative tweets labelled informative (87). CT-BERT-1 cut that number to 52. The remaining errors mixed statistics with opinion, personal stories, politics or protective measures.

Interesting ideas

• I1. Replacing all numbers with one placeholder while sparing the domain-meaningful "19" in "COVID-19" lets n-gram models generalize over figures without destroying a name.
• I2. Before adding a feature, the authors plotted its distribution and found the data was already almost entirely neutral in sentiment, so the feature could not discriminate.
• I3. Giving earlier transformer layers smaller learning rates (decay factor 0.95 per layer) protects general pretrained knowledge while the top layers adapt to the task.
• I4. Hiding the real test items inside a much larger pool stops participants from tuning on the test set.
• I5. The authors note that a true, informative message may still be stale, such as yesterday's case count, so validity depends on time.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): The en tokenizer could offer a number-abstraction view with a whitelist of numeric tokens that belong to names, keeping both the raw value and the class for later stages.
• I2 (learn): Run a quick distribution check on each candidate feature or repair-rule trigger in silver data, and drop those that are near-constant before spending training effort.
• I3 (tree): gparse fine-tuning on a new treebank or UD dialect could apply layer-wise decay to keep low-level syntactic representations stable.
• I4 (learn): Evaluation of readers or parsers could embed gold items among unlabeled decoys, a cheap negative control against overfitting to known benchmarks.
• I5 (world): Facts and quantities in the text world could carry a reference time, so answers can flag outdated states instead of treating every reported number as current.