📄 R00 at NLP4IF-2021 Fighting COVID-19 Infodemic with Transformers and More Transformers

Summary

The paper presents the top-ranked system in the Arabic track of the NLP4IF-2021 shared task on the COVID-19 infodemic. Each tweet receives seven yes/no labels, covering questions such as whether it holds a verifiable factual claim, seems false, is harmful, needs a professional fact-checker, or deserves government attention. Questions 2–5 only make sense when the first answer is "yes". The dataset is small and heavily skewed: 2,536 training, 520 development and 1,000 test tweets.

The authors treat the task as multi-label classification. They fine-tune several Arabic BERT-style encoders (AraBERT, Asafaya-BERT, ARBERT, MARBERT), each with a linear head, on both raw and cleaned versions of the text. Five of these models are then merged by a weighted mean of their sigmoid outputs, with a 0.5 threshold. The ensemble reached F1 0.780 and accuracy 0.762 on the test set and took first place.

The work matters because it shows two things. First, the same text cleaning can help or hurt depending on each model's pretraining corpus. Second, mixing a strong generalist with label-specific models is a cheap way to lift rare labels.

Key points

• Labels and post-processing:
  • Seven questions are asked per tweet.
  • For training, "NaN" labels on Q2–Q5 were mapped to "No".
  • At prediction time, Q2–Q5 are forced back to "NaN" whenever the model answers Q1 with "No".
• Training setup:
  • Maximum sequence length is 100; batch sizes are 32 for training and 16 for validation.
  • Models ran for 20 epochs, keeping the checkpoint with the best weighted F1. Scores mostly flattened after about 10 epochs.
  • Learning rates were chosen by hand from trial runs, not systematically tuned.
• Effect of cleaning:
  • Cleaning removed stop-words, numbers, punctuation and non-Arabic tokens (NLTK); the farasapy-based Arabert-Preprocessor was used for AraBERT only.
  • Cleaning noticeably helped the MSA-oriented ARBERT (weighted F1 0.715 → 0.734).
  • Asafaya-BERT, whose pretraining includes dialectal text, did best on raw tweets (0.750 for both Base and Large).
• Per-label difficulty: label difficulty varies sharply on validation.
  • Q2 (False Information) F1 stays between 0.10 and 0.26.
  • Q7 (needs government attention) sits around 0.83–0.85.
  • Exact-match accuracy of single models is only about 0.34–0.44.
• Final ensemble:
  • Five models trained on cleaned data, with weights 3, 3, 1, 1 and 3.
  • Validation scores: weighted F1 0.749, micro F1 0.763, accuracy 0.405.
  • The test score appears as 0.781/0.763 in the results section and 0.780/0.762 in the abstract.
  • The model names in the ensemble list do not fully match the table IDs.

Interesting ideas

• I1. Encode the dependency between questions as a hard rule at decoding time: child labels are switched off unless the parent label fires, so outputs never contradict each other.
• I2. Combine one strong all-round model with several specialists for weak labels through a weighted vote. The generalist carries most of the weight, while the specialists only tip the balance on the labels they are good at.
• I3. Test preprocessing separately for each model, because normalisation that helps a model trained on formal text can strip useful signal from a model trained on informal text.
• I4. Report per-label F1 next to the aggregate scores and the strict all-labels-correct accuracy. This exposes how a respectable average (about 0.75 F1) can hide near-useless labels and low exact-match accuracy (about 0.4).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): only run checks like "is this claim false/harmful?" on clauses already classified as assertions, and leave the rest explicitly marked as not applicable.
• I2 (learn): when witness parsers vote on silver trees, give a reliable general parser a larger weight and let specialists for rare relations nudge only those arcs.
• I3 (tokens): evaluate tokenizer and normaliser options (e.g., handling of hashtags, emojis, casing) against each downstream consumer, rather than fixing one cleaning pipeline for everyone.
• I4 (answer): score school-test readers per question type and by whole-item exact match, so that weak categories are not hidden by the mean.