📄 Not-NUTs at W-NUT 2020 Task 2: A BERT-based System in Identifying Informative COVID-19 English Tweets

Summary

The paper addresses the W-NUT 2020 Task 2 shared task. The goal is to decide whether an English tweet about COVID-19 is informative, meaning it reports suspected, confirmed, recovered or fatal cases, or where those cases were and travelled. The authors use the organisers' 10,000-tweet dataset. Their core encoder is BERTweet, a Transformer pre-trained on tweets, topped with a small stack of linear layers. They compare several ways of building the sentence vector from different BERTweet layers. They also propose a "Global Local" variant: one encoder reads the whole tweet while two others read its first and second halves, and the three vectors are joined. Training runs in two stages: the head learns first on a frozen encoder, then the whole network is fine-tuned. Pooling many configurations by majority vote gave an F1 of 0.8991 on the hidden test set, placing 12th of 56 teams and roughly a point behind the winner. This shows that a tweet-specific encoder plus simple ensembling is competitive for crisis-information filtering.

Key points

• Dataset: 10,000 COVID-19 tweets, each labelled by 3 annotators (Fleiss' Kappa 0.818) and split 70/10/20. For the final run, train and validation were merged and re-split 90/10.
• Baselines: tf-idf features with classic classifiers reached validation F1 of 0.7827 (Logistic Regression), 0.7678 (SVM) and 0.7486 (Naive Bayes).
• Layer combinations: using only the last BERTweet layer gave F1 0.8912. Concatenating the last and first layers gave the best single result in that table, 0.9045.
• Global Local BERTweet and ensembling: the best Global Local setup reached 0.9075. Ensembling all models scored 0.9130 with majority voting and 0.9111 with softmax averaging.
• Length-aware ensemble (post-competition): tweets were grouped as short (0–22 words), medium (23–44) or long (>44). For each group, the 7 models with the most correct training predictions were used, which raised validation F1 to 0.9182.

Interesting ideas

• I1. Mixing a shallow layer with the top layer of the encoder beat using the top layer alone, which suggests low-level surface signals still carry useful information after fine-tuning.
• I2. A separate set of ensemble members was picked for each input-length band, based on which models were most often right on that band.
• I3. Besides reading the whole text, the system also encodes its two halves separately and merges all three views, so local detail is not drowned out in long inputs.
• I4. Training runs in two stages: the new head is trained with the pretrained encoder frozen, and only then is everything fine-tuned.
• I5. The paper compares hard majority voting with averaging of probability vectors and finds hard voting slightly ahead in this setting.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could feed its arc and label scorers a concatenation of an early and the final Transformer layer, then test whether this helps on noisy or informal input.
• I2 (learn): when several witness parsers vote on silver trees, their weight could depend on how accurate each one has been for sentences of similar length, rather than staying fixed.
• I3 (tree): for long sentences, gparse could combine full-sentence context with encodings of each half, which might help attachments far from the root.
• I4 (learn): new heads added to gparse, such as a new label set after UD dialect conversion, could be warmed up on frozen features first to avoid disturbing the pretrained weights early on.
• I5 (learn): the witness-parser agreement step could be evaluated both ways, counting votes per arc versus averaging arc probabilities, to pick the rule that yields cleaner silver data.