Elamur Research⌕
☰
home / papers / 2021 / Detecting Multilingual COVID-19 Misinformation on Social Media via Contextualized Embeddings

Detecting Multilingual COVID-19 Misinformation on Social Media via Contextualized Embeddings

Paper · 2021 · Subhadarshi Panda and Sarah Ita Levitan

Summary

The authors build classifiers that flag COVID-19 misinformation in tweets written in English, Bulgarian and Arabic. They entered these systems in the NLP4IF 2021 shared task "Fighting the COVID-19 Infodemic". Each tweet carries seven binary annotations, such as whether it makes a verifiable claim or appears false. The authors predict all seven jointly with four multitask architectures: a bag-of-embeddings logistic regression, a transformer encoder trained from scratch, English BERT and multilingual BERT. Each model has a separate linear head per question, and training minimizes the sum of the per-question losses. To study cross-lingual transfer, they vary how much target-language data enters training: none, small random samples, all of it combined with source data, or target data alone. m-BERT transfers best and stays strong even with zero target-language examples. The final systems use English BERT for English and m-BERT for the other two languages. They placed second for Bulgarian and Arabic and were competitive for English. The work suggests that misinformation detectors can be extended to languages with little labeled data.

Key points

  • The data are tweets from the shared task (Shaar et al., 2021), labeled Yes/No on seven questions (Q1–Q7). Q2–Q5 are "nan" whenever Q1 is No. Training sets are small: 451 English, 3000 Bulgarian and 198 Arabic tweets, before the additional data was released.
  • All four models share one design: a [CLS] or summed representation feeds seven linear heads, and the cross-entropy losses are added. For BERT, keeping the pretrained weights frozen gave better results than updating them. Training used early stopping after 10 epochs without dev F1 improvement.
  • In the zero-shot setting, m-BERT reached 0.81 dev F1 on Bulgarian when trained only on English, and 0.808 when trained only on Arabic. It reached 0.685 on English and 0.672 on Arabic.
  • Test F1 scores were 0.736 for English (bert-base-uncased), 0.817 for Bulgarian and 0.741 for Arabic (both bert-base-multilingual-cased).
  • The late-released extra data had mixed effects. It raised Arabic dev F1 from 0.556 to 0.688 but did not help English.

Interesting ideas

  • I1. After prediction, a rule-based pass enforces the dependency between labels: when the parent question is answered No, the child answers are overwritten so the output is always structurally valid.
  • I2. Instead of training separate classifiers, the authors train one shared encoder with a cheap head per question and a summed loss, so related judgements share the same representation.
  • I3. To build a learning curve, the authors add a fixed number of randomly sampled target-language examples (50, 100, 150, 200) to the source data. This shows how few in-domain labels are needed before performance levels off.
  • I4. On small training sets, freezing the pretrained encoder and training only the added head worked better than full fine-tuning.
  • I5. The authors check whether newly added training data actually helps on the dev set rather than assuming it does, and the effect differed by language.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): when prag fills several interdependent slots for an utterance, a final consistency pass could blank slots that a parent decision rules out. Each such override would be logged as an explicit, explainable step.
  • I2 (frames): speech-act type and related pragmatic flags could be predicted from one sentence representation with per-flag heads, keeping the model small while each output stays separately inspectable.
  • I3 (learn): when adapting the parser to a new genre such as fables or maths problems, the same sweep would show how many gold trees are worth annotating before silver data takes over.
  • I4 (tree): for gparse on a small or converted treebank, freezing the transformer and training only the arc and label scorers is a cheap baseline to try first.
  • I5 (learn): every batch of new silver or teacher-labeled data should pass a before/after dev comparison per domain, alongside the negative controls, before being kept.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hunter CollegeNew York · United States9 works · 1,245 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksInformation Systems · 1,730 worksSociology and Political Science · 626 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksMultilingual & cross-lingual NLP · 882 works

Cite

Cite as Subhadarshi Panda and Sarah Ita Levitan. 2021. Detecting Multilingual COVID-19 Misinformation on Social Media via Contextualized Embeddings. Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.nlp4if-1.19

@inproceedings{panda2021detecting,
  title = {{Detecting Multilingual COVID-19 Misinformation on Social Media via Contextualized Embeddings}},
  author = {Subhadarshi Panda and Sarah Ita Levitan},
  booktitle = {Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda},
  year = {2021},
  doi = {10.18653/v1/2021.nlp4if-1.19},
  url = {https://doi.org/10.18653/v1/2021.nlp4if-1.19}
}

References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pasquale Capuozzo, Ivano Lauriola, Carlo Strapparava, Fabio Aiolli, and Giuseppe Sartori. 2020a. Automatic detection of cross-language verbal deception. find
  2. Pasquale Capuozzo, Ivano Lauriola, Carlo Strapparava, Fabio Aiolli, and Giuseppe Sartori. 2020b. De-cOp: A multilingual and multi-domain corpus for detecting deception in typed text. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1423–1430, Marseille, France. European Language Resources Association. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  4. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. ↗
  5. Tamanna Hossain, Robert L Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. Covidlies: Detecting covid-19 misinformation on social media. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020.
  6. Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations. ↗
  7. Tanushree Mitra, Graham P Wright, and Eric Gilbert. 2017. A parsimonious language model of social media credibility across disparate events. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing, pages 126–145. find
  8. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc. ↗
  9. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics. ↗
  10. Shaden Shaar, Firoj Alam, Giovanni Da San Martino, Alex Nikolov, Wajdi Zaghouani, Preslav Nakov, and Anna Feldman. 2021. Findings of the NLP4IF-2021 shared task on fighting the COVID-19 infodemic and censorship detection. In Proceedings of the Fourth Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, NLP4IF@NAACL’ 21, Online. Association for Computational Linguistics. find
  11. Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social media: A data mining perspective. ACM SIGKDD explorations newsletter, 19(1):22–36. ↗
  12. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  13. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗
  14. Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 19–27.
  15. Verónica Pérez-Rosas and Rada Mihalcea. 2014. Cross-cultural deception detection. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 440–445, Baltimore, Maryland. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/panda-2021-detecting-multilingual-covid-19-misinformation/