📄 Not-NUTs at WNUT-2020 Task 2: A BERT-based System in Identifying Informative COVID-19 English Tweets

Summary

The paper tackles WNUT-2020 Task 2: deciding whether an English COVID-19 tweet is informative, meaning it reports suspected, confirmed, recovered or fatal cases, or the location and travel history of those cases. The authors use the shared-task corpus of 10,000 tweets and build on BERTweet, a Transformer pre-trained on tweets, with a small stack of linear layers as the classification head. They compare several ways of pooling the [CLS] representation across Transformer layers. They also introduce a "Global Local BERTweet" that trains three encoders together: one reads the whole tweet, one the first part and one the rest. Training runs in two stages, first fitting only the head and then fine-tuning the full network. Many configurations are then combined by majority voting or by averaging softmax outputs. The submitted ensemble reached an F1 of 0.8991 on the hidden test set, placing 12th of 56 teams and about one point behind the winner. The work shows that cheap layer-mixing, part-of-text encoders and ensembling add measurable gains over a single fine-tuned domain model.

Key points

• The data is 10,000 tweets, each labelled by 3 annotators, with a Fleiss' Kappa of 0.818. It is split 70/10/20 into train, validation and test. For the final run, train and validation were merged and re-split 90/10.
• tf-idf baselines built with sklearn scored F1 0.7827 for Logistic Regression, 0.7678 for SVM and 0.7486 for Naive Bayes on validation.
• Using only BERTweet's last layer gave F1 0.8912. Concatenating the last and first layers gave the best single-encoder result at 0.9045. Global Local variants reached up to 0.9075.
• On validation, majority voting scored F1 0.9130 and softmax averaging 0.9111. The test result was 0.8991, against 0.9096 for the top team.
• After the competition, the authors grouped tweets by length: short (0–22 words), medium (23–44) and long (>44). For each group they picked the 7 models with the most correct training predictions. This raised validation F1 to 0.9182.

Interesting ideas

• I1. The text is encoded both as a whole and as separate segments, and the pooled vectors are concatenated, so that locally concentrated cues are not diluted by the rest of the sequence.
• I2. Representations from shallow and deep layers are mixed (for example first + last) instead of relying only on the top layer. This helped because lower layers keep surface and lexical signals that the top layer partly loses.
• I3. Training runs in two stages: the pre-trained encoder is frozen while a fresh head is fitted, then everything is unfrozen at a lower learning rate. This keeps the head's random start from damaging the pre-trained weights.
• I4. Ensemble members are chosen per input-length bucket, keeping the models that perform best on that range of lengths. This treats input length as a factor that decides which model to trust.
• I5. Cheap tf-idf baselines are reported next to the neural system, which shows how much the expensive model actually adds.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): when assigning speech acts or detecting the newsworthy clause, the analysis could score the main clause and the trailing segments separately, then combine the scores while keeping track of which segment drove the decision.
• I2 (tree): gparse could feed its arc scorer a concatenation of an early and a late transformer layer, and test on a held-out treebank whether LAS improves.
• I3 (learn): when new heads such as role or frame classifiers are attached to a pretrained parser encoder, warming up the head first should make training on silver data more stable.
• I4 (learn): evaluation could report parser and reader accuracy broken down by sentence length. A length-aware choice among witness parsers could then decide whose analysis wins when building silver data.
• I5 (learn): every new component could be compared against a trivial bag-of-words control. This serves as a negative control to confirm that structural reasoning, not lexical overlap, explains the score.