📄 IRLab@IITBHU at WNUT-2020 Task 2: Identification of informative COVID-19 English Tweets using BERT

Summary

The paper describes a system for W-NUT 2020 Task 2, which asks whether an English tweet about COVID-19 is informative or not. The organizers supplied 10K tweets, each labeled by three annotators. The authors cleaned the text by removing hashtag signs, user mentions, non-ASCII symbols, emoticons and numbers. They also dropped stopwords, except for the transformer models. They then compared several tweet representations: Word2Vec trained on the task data, pre-trained GloVe and FastText vectors, and sentence vectors from bert-base-uncased and distilbert-base-uncased. These were paired with several classifiers: Logistic Regression, SVM, XGBoost, BiLSTM and GRU. The two systems with the best validation scores were submitted. Sentence embeddings from DistilBERT fed into Logistic Regression won, with an F1 of 0.7508 on the informative class of the test set. FastText with GRU came close behind. The study is a broad, simple comparison of off-the-shelf options for noisy crisis tweets. It shows that a compact pre-trained encoder with a linear classifier on top beats heavier recurrent pipelines built on static word vectors.

Key points

• The dataset (Nguyen et al., 2020) has 10K tweets: 4719 INFORMATIVE and 5281 UNINFORMATIVE. It is split 7000/1000/2000 into train, validation and test, and annotator agreement is Fleiss' Kappa 0.818.
• The best submission was DistilBERT + LR: F1 .7508, precision .7904, recall .7150, accuracy .7760. FastText + GRU reached F1 .7361.
• The official metrics are computed only for the INFORMATIVE class. The authors' own validation table covers both classes, so the two sets of numbers differ somewhat.
• GRU outperformed BiLSTM. FastText outperformed Word2Vec and GloVe, which the authors attribute to its character n-gram subwords helping with rare words.
• Some tokens, such as HTTPURL and "coronavirus", appear heavily in both classes. The authors name these as a source of confusion. They list fine-tuning the pre-trained models as future work, since here the models were used only as feature extractors.

Interesting ideas

• I1. A frozen pre-trained encoder with only a logistic regression on top beat the more complex trained recurrent models, making it a cheap and strong reference point.
• I2. Character n-gram subwords give usable vectors even for rare or misspelled words, because those words share pieces with common ones.
• I3. The authors spotted tokens that occur often in both classes (HTTPURL, "coronavirus") as cues that mislead the models rather than help them.
• I4. Preprocessing was tailored per model: stopwords were removed for the classic models but kept for the BERT-family encoders, which rely on full context.
• I5. The paper shows how scoring only the positive class makes official results diverge from figures averaged over both classes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Any new explainable component could be benchmarked against a "frozen encoder + linear probe" baseline, which shows how much the structured pipeline adds over a minimal opaque system.
• I2 (tokens): Subword or affix evidence could back up the tagger and lemmatizer when they meet unknown or noisy word forms, while keeping the guess traceable to the shared affixes.
• I3 (learn): Silver data and repair rules could be screened for features equally frequent across labels, flagging them as possible shortcuts in negative-control evaluations.
• I4 (tokens): This supports keeping all function words intact before UD parsing and frame extraction, since stripping them would destroy the cues a structural pipeline depends on.
• I5 (answer): Test reports for the readers should state which class or subset each metric covers, so that scores from different evaluations are not compared like for like by mistake.