Elamur Research⌕
☰
home / papers / 2021 / Transformers to Fight the COVID-19 Infodemic

Transformers to Fight the COVID-19 Infodemic

Paper · 2021 · Lasitha Uyangodage, Tharindu Ranasinghe and Hansi Hettiarachchi

Summary

The paper describes the InfoMiner team's entry to the NLP4IF-2021 shared task on the COVID-19 infodemic. The task asked systems to label tweets in English, Arabic and Bulgarian with seven yes/no properties: whether a tweet holds a verifiable factual claim, seems false, matters to the public, is harmful, needs a fact-checker, harms society, or deserves government attention. The authors treat this as seven separate binary problems and fine-tune one pretrained transformer per property, using the [CLS] representation with a softmax layer on top. The data is heavily skewed, which pushes models toward always predicting the majority label, so they shrink the majority class until it matches the minority class. They also skip language-specific preprocessing so that one recipe serves all three languages, and they combine runs from several random seeds by majority vote. The submission reached mean F1 scores of 0.864 for English, 0.707 for Arabic and 0.578 for Bulgarian, placing 4th in every language. The work shows that a plain, uniform transformer pipeline can stay competitive across languages, and the code is public (infominer on GitHub).

Key points

  • Models compared on the development set:
    • English: bert-base-cased (mean macro F1 0.587) beat roberta-base (0.523).
    • Arabic: arabert-v2-tokenized (0.606) beat arabert-v2 (0.583) and bert-multilingual-cased (0.536). The tokenized variant uses Farasa pre-segmentation of prefixes and suffixes.
    • Bulgarian: time allowed only bert-multilingual-cased (0.375).
  • Training setup:
    • The majority class was undersampled to the minority count, then split 0.8:0.2 into train and validation.
    • The best learning rate was 1e-5 and the best epoch count was 3, the same for all languages and labels.
    • Other settings: batch size 8, Adam, 10% linear warm-up, and early stopping after ten evaluation rounds without improvement.
    • Training ran on a Tesla K80.
  • Each configuration was run with five random seeds, and the final label was the majority vote across those runs.
  • Test results compared with the best systems:
    • English: 0.864 vs 0.897.
    • Arabic: 0.707 vs 0.781. On label I in Arabic, InfoMiner (0.852) slightly beat the best system (0.843).
    • Bulgarian: 0.578 vs 0.837.
    • The random baseline was about 0.496 everywhere, though InfoMiner's Bulgarian label III score (0.419) fell below it.
  • Stated limitation: running seven models is expensive. The authors propose a single multilabel model as future work.

Interesting ideas

  • I1. Training the same model with several random seeds and keeping the label most runs agree on is a cheap way to steady predictions from unstable fine-tuning.
  • I2. When a binary property is rare, cutting the frequent class down to the rare class's size keeps the classifier from simply always answering "no".
  • I3. Splitting words into prefixes and suffixes (Farasa) before the subword model helped Arabic more than the raw-text version of the same model. This suggests explicit morphological segmentation adds value even beneath a strong pretrained encoder.
  • I4. Reporting per-label scores next to a random baseline shows where a system is truly learning and where it is near chance; the Bulgarian label III result below random would vanish inside a mean score.
  • I5. The paper weighs one model per property, which is flexible to tune, against one shared multilabel model, which is cheaper to maintain, and names cost as the deciding factor.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Seed-voting could be added as an extra "witness" next to the different parsers when building silver data, so arcs that flip between seeds get flagged as unreliable.
  • I2 (frames): Rare pragmatic labels like irony or particular speech acts could be trained on balanced subsets so the detector does not collapse to the default reading.
  • I3 (tokens): Feeding gparse lemma and affix-split input from crate en, alongside surface forms, could be tested as a cheap boost for rare or inflected words.
  • I4 (learn): Evaluation reports should list each frame or label type against a random or shuffled control, so components that only look good on average get caught.
  • I5 (answer): When adding test-specific readers, it helps to decide up front whether each question type gets its own module or one shared head, keeping maintenance cost in view.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WolverhamptonWolverhampton · United Kingdom35 works · 6,331 citations🎓University of MünsterMünster · Germany4 works · 854 citations🎓Birmingham City UniversityBirmingham · United Kingdom5 works · 26 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Lasitha Uyangodage, Tharindu Ranasinghe and Hansi Hettiarachchi. 2021. Transformers to Fight the COVID-19 Infodemic. Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.nlp4if-1.20

@inproceedings{uyangodage2021transformers,
  title = {{Transformers to Fight the COVID-19 Infodemic}},
  author = {Lasitha Uyangodage, Tharindu Ranasinghe and Hansi Hettiarachchi},
  booktitle = {Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda},
  year = {2021},
  doi = {10.18653/v1/2021.nlp4if-1.20},
  url = {https://doi.org/10.18653/v1/2021.nlp4if-1.20}
}

References

27 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California. Association for Computational Linguistics.
  2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  3. Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information credibility on twitter. In Proceedings of the 20th International Conference on World Wide Web, WWW ’11, page 675–684, New York, NY, USA. Association for Computing Machinery. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  5. Ibrahim Abu El-khair. 2016. 1.5 billion words arabic corpus. In arXiv preprint arXiv:1611.04033. ↗
  6. Bin Guo, Yasan Ding, Lina Yao, Yunji Liang, and Zhiwen Yu. 2020. The future of false information detection on social media: New perspectives and trends. ACM Comput. Surv., 53(4). ↗
  7. Hansi Hettiarachchi and Tharindu Ranasinghe. 2019. Emoji powered capsule network to detect type and target of offensive posts in social media. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 474–480, Varna, Bulgaria. INCOMA Ltd. find
  8. Hansi Hettiarachchi and Tharindu Ranasinghe. 2020a. BRUMS at SemEval-2020 task 3: Contextualised embeddings for predicting the (graded) effect of context in word similarity. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages Hansi Hettiarachchi and Tharindu Ranasinghe. 2021. TransWiC at SemEval-2021 Task 2: Transformer-based Multilingual and Cross-lingual Word-in-Context Disambiguation. In Proceedings of the Fifteenth Workshop on Semantic Evaluation. find
  9. Tommi Jauhiainen, Tharindu Ranasinghe, and Marcos Zampieri. 2021. Comparing approaches to Dravidian language identification. In Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 120–127, Kiyv, Ukraine. Association for Computational Linguistics. ↗
  10. Simon Kemp. 2021. 15.5 users join social every second (and other key stats to know). https://blog.hootsuite.com/ simon-kemp-social-media/. ↗
  11. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In arXiv preprint arXiv:1412.6980. ↗
  12. S. Kwon, M. Cha, K. Jung, W. Chen, and Y. Wang. 2013. Prominent features of rumor propagation in online social media. In 2013 IEEE 13th International Conference on Data Mining, pages 1103– 1108. find
  13. Quanzhi Li, Qiong Zhang, and Luo Si. 2019. Rumor detection by exploiting user credibility information, attention and multi-task learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1173–1179, Florence, Italy. Association for Computational Linguistics. find
  14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. In arXiv preprint arXiv:1907.11692. ↗
  15. Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J. Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI’16, page 3818–3824. AAAI Press. find
  16. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. ACM Trans. Internet Technol., 17(3). ↗
  17. Azzam Mourad, Ali Srour, Haidar Harmanai, Cathia Jenainati, and Mohamad Arafeh. 2020. Critical impact of social networks infodemic on defeating coronavirus covid-19 pandemic: Twitter-based study and research directions. IEEE Transactions on Network and Service Management, 17(4):2145–2155. ↗
  18. Zesis Pitenis, Marcos Zampieri, and Tharindu Ranasinghe. 2020. Offensive language identification in Greek. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5113– 5119, Marseille, France. European Language Resources Association. ↗
  19. Vahed Qazvinian, Emily Rosengren, Dragomir R. Radev, and Qiaozhu Mei. 2011. Rumor has it: Identifying misinformation in microblogs. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1589–1599, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  20. Tharindu Ranasinghe, Sarthak Gupte, Marcos Zampieri, and Ifeoma Nwogu. 2020. WLV-RIT at HASOC-Dravidian-CodeMix-FIRE2020: Offensive Language Identification in Code-switched YouTube Comments. In Proceedings of the 12th annual meeting of the Forum for Information Retrieval Evaluation. ↗
  21. Tharindu Ranasinghe and Hansi Hettiarachchi. 2020. BRUMS at SemEval-2020 task 12: Transformer based multilingual offensive language identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1906– 1915, Barcelona (online). International Committee for Computational Linguistics. find
  22. Tharindu Ranasinghe, Diptanu Sarkar, Marcos Zampieri, and Alex Ororbia. 2021. WLV-RIT at SemEval-2021 Task 5: A Neural Transformer Framework for Detecting Toxic Spans. In Proceedings of the Fifteenth Workshop on Semantic Evaluation. find
  23. Tharindu Ranasinghe and Marcos Zampieri. 2020. Multilingual offensive language identification with cross-lingual embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5838–5844, Online. Association for Computational Linguistics.
  24. Shaden Shaar, Firoj Alam, Giovanni Da San Martino, Alex Nikolov, Wajdi Zaghouani, Preslav Nakov, and Anna Feldman. 2021. Findings of the NLP4IF-2021 shared task on fighting the COVID-19 infodemic and censorship detection. In Proceedings of the Fourth Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, NLP4IF@NAACL’ 21, Online. Association for Computational Linguistics. find
  25. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? In Chinese Computational Linguistics, pages 194– 206, Cham. Springer International Publishing. ↗
  26. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with BERTserini. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 72–77, Minneapolis, Minnesota. Association for Computational Linguistics.
  27. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182, Florence, Italy. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/uyangodage-2021-transformers-fight-covid-19-infodemic/