Elamur Research⌕
☰
home / papers / 2021 / DamascusTeam at NLP4IF2021: Fighting the Arabic COVID-19 Infodemic on Twitter Using AraBERT

DamascusTeam at NLP4IF2021: Fighting the Arabic COVID-19 Infodemic on Twitter Using AraBERT

Paper · 2021 · Ahmad Hussein, Nada Ghneim and Ammar Joukhadar

Summary

The paper describes a system for the NLP4IF 2021 shared task on the COVID-19 infodemic. Given an Arabic tweet about the pandemic, the system answers seven yes/no questions. The questions ask whether the tweet makes a checkable claim, seems false, matters to the public, is harmful, needs fact-checking, harms society, or needs government attention. The authors used the 2,556 labelled tweets released by the organizers.

Their pipeline has two stages. First, tweets are cleaned: the ekphrasis tool replaces noisy items such as numbers, dates, URLs-like constructs and phone numbers with generic placeholder forms. Second, a cased AraBERTv0.2-base model is fine-tuned, with one classifier per property built on the same architecture. Classification uses the [CLS] vector, a fully connected layer and softmax.

On their own held-out split, accuracy ranged from roughly three-quarters to above nine-tenths, depending on the question. On the organizers' blind test the system scored an F1 of 0.664, placing 5th. The work shows that a standard pretrained Arabic encoder plus light normalization gives a competitive baseline for multi-aspect tweet triage.

Key points

  • The data is the NLP4IF 2021 Arabic set (Shaar et al., 2021): 2,556 tweets, each labelled for seven binary properties Q1–Q7. Labels for Q2–Q5 are left as "nan" whenever Q1 (verifiable claim) is No.
  • The model is AraBERTv0.2-base, cased, with 12 layers, 12 heads, hidden size 768 and about 136M parameters. It was pre-trained on roughly 77 GB drawn from five corpora (OSCAR, Arabic Wikipedia, the 1.5B words corpus, OSIAN, Assafir).
  • Training settings were:
    • maximum sequence length cut to 128 because tweets are short;
    • batch size 6 with 16 gradient accumulation steps;
    • 10 epochs;
    • learning rate 2e-5;
    • cross-entropy loss.
  • The internal split was 90/10 train/test, and the training part was split again 90/10 for validation. Held-out accuracy was:
    • Q7: 92%
    • Q6: 90%
    • Q2: 86%
    • Q3: 84%
    • Q4: 76%
    • Q5: 75%
    • Q1: 74%
  • The blind test was scored by the organizers: precision 78.7%, recall 67.7%, accuracy 67.7%, F1 66.4%, placing 5th in the task.

Interesting ideas

  • I1. Noisy surface items (numbers, dates, money amounts, percentages, times, phone numbers, emails) are swapped for class placeholders, because the specific value seldom matters for the label and only adds sparsity.
  • I2. The label scheme is hierarchical: the follow-up questions are defined only when the first question (does it contain a verifiable claim) is answered Yes. This gates fine-grained judgments behind a coarse one.
  • I3. The input length limit was shrunk from 512 to 128 to match the short texts, trading unused capacity for speed and memory.
  • I4. Small batches combined with gradient accumulation give an effectively large batch on modest hardware.
  • I5. Results are reported both on a self-made split and on an external blind test, and the blind F1 is clearly lower than the internal per-question figures, which exposes optimism in in-house evaluation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): the tokenizer could emit typed placeholder tokens for such spans, keeping the raw value attached as an attribute so later stages can still reason about quantities.
  • I2 (frames): speech-act analysis could first decide whether a clause is an assertion at all, and only then compute truth-related properties like checkability, leaving them explicitly undefined otherwise.
  • I3 (tree): gparse could bucket or cap sequence length per genre (short sentences, tweets, fable lines) to train and run faster without losing coverage.
  • I4 (learn): GPU training runs for gparse could use the same trick to stabilize updates when memory limits per-step batch size.
  • I5 (learn): every evaluation report could pair internal dev numbers with a held-out set no one tuned on, treating the gap as a health metric alongside negative controls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ahmad Hussein, Nada Ghneim and Ammar Joukhadar. 2021. DamascusTeam at NLP4IF2021: Fighting the Arabic COVID-19 Infodemic on Twitter Using AraBERT. Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.nlp4if-1.13

@inproceedings{hussein2021damascusteam,
  title = {{DamascusTeam at NLP4IF2021: Fighting the Arabic COVID-19 Infodemic on Twitter Using AraBERT}},
  author = {Ahmad Hussein, Nada Ghneim and Ammar Joukhadar},
  booktitle = {Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda},
  year = {2021},
  doi = {10.18653/v1/2021.nlp4if-1.13},
  url = {https://doi.org/10.18653/v1/2021.nlp4if-1.13}
}

References

25 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Heejung Jwa, Dongsuk Oh, Kinam Park, Jang Mook Kang, and Heuiseok Lim. 2019. exBA KE: Automatic fake news detection model based on Bidirectional Encoder Representations fro m Transformers (BERT). In Applied Sciences (Switzerland), 9(19), [4062] . find
  2. Hadeer Ahmed, Issa Traore, Sherif Saad. 2017. Detection of online fake news using N-gram analysis and machine learning techniques. In International conference on intelligent, secure, and dependable systems in distributed and cloud environments. Springer, Cham, pages 127–138. find
  3. Srijan Kumar, and Neil Shah.2018. False Informatio n on Web and Social Media: A Survey. In arXiv:arXiv-1804. find
  4. Hunt Allcott, and Matthew Gentzkow. 2017. Social Media and Fake News in the 2016 Election. In Journal of Economic Perspectives. 31 (2): 211-36. ↗
  5. Athanasia Ntalla, and Stavros T. Ponis. (2015). Twitter as an instrument for crisis response: The Typhoon Haiyan case study. In The 12th International Conference on Information Systems for Crisis Response and Management. find
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based Model for Arabic Language Understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference. pages 11-16. ↗
  7. Natali Ruchansky, Sungyong Seo, and Yan Liu. 2017. CSI: A Hybrid Deep Model for Fake News Detection. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM '17). Association for Computing Machinery, New York, NY, USA, pages 797–806. ↗
  8. Christos Baziotis, Nikos Pelekis, and Christos Doulkeridis. 2017. DataStories at SemEval-20 1 7 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016. San Diego, CA, USA, 16–17 June 2016; pages 747–754. find
  9. Shaden Shaar, Firoj Alam, Giovanni Da San Martino, Alex Nikolov, Wajdi Zaghouani, Preslav Nakov,and Anna Feldman. 2021. Findings of the {NLP4IF} -2021 Shared Task on Fighting the {COVID}- 1 9 Infodemic and Censorship Detection. In Proceedings of the Fourth Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda. find
  10. Alessandro Bondielli, and F. Marcelloni. 2019. A survey on fake news and rumour detection techniques. In Inf. Sci. 497. pages 38-55. ↗
  11. Linda Cappellato, Carsten Eickhoff, Nicola Ferro, and Aurélie Névéol. 2020. CLEF 2020 Working Notes. In CEUR Workshop Proceedings, CEUR-WS.org. find
  12. Jieun Shin, Lian Jian, Kevin Driscoll, and Franois Bar. 2018. The diffusion of misinformation on social media. Comput. Hum. Behav. 83, C (June 2018), pages 278–287. find
  13. Souvick Ghosh, and Chirag Shah. 2018. Towards automatic fake news classification. Proceedings of the Association for Information Science and Technology. 55(1): pages 805–807. ↗
  14. Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, Huan Liu. 2020. FakeNewsNet: A Data Repository with News Content, Social Context, and Spatiotemporal Information for Studying Fake News on Social Media. In Big Data. 8. 171-188. 10.1089/big.2020.0062. ↗
  15. Genevieve Gorrell, Elena Kochkina, Maria Liakata, Ahmet Aker, Arkaitz Zubiaga, Kalina Bontcheva, and Leon Derczynski. 2019. SemEval-2019 task 7: RumourEval, determining rumour veracity and support for rumours. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 845–854.
  16. Kai Shu, Limeng Cui, Suhang Wang, Dongwon Lee, and Huan Liu. 2019. Defend: Explainable fake news detection. In KDD 2019 - Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pages 395-405. find
  17. Maram Hasanain, Fatima Haouari, Reem Suwaileh , Zien Sheikh Ali,Bayan Hamdan, Tamer Elsayed , Alberto Barrón-Cedeňo, Giovanni Da San Martino , and Preslav Nakov. 2020. Overview of CheckThat! 2020 Arabic: Automatic Identi-fication and Verification of Claims in Social Media. In arXiv:2007.07997. ↗
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukas z Kaiser, Illia Polosukhin. 2017. Attention is All you Need. In Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017.
  19. Ahmad Hussein, Abdulkarim Hussein, Nada Ghneim, and Ammar Joukhadar. 2020. DamascusTeam at CheckThat! 2020: Check worthiness on Twitter with hybrid CNN and RNN models. In Cappellato et al. (2020). find
  20. Michela Del Vicario, Alessandro Bessi, Fabiana Zollo, Fabio Petroni, Antonio Scala, Guido Caldarelli, H. Eugene Stanley, and Walter Quattrociocchi. 2016. find
  21. The spreading of misinformation online. In Proceedings of the National Academy of Sciences Jan 2016, 113 (3) pages 554-559; DOI: 10.1073/pnas.1517441113. ↗
  22. Soroush Vosoughi, Mostafa ‘Neo’ Mohsenvand, and Deb Roy. 2017. Rumor Gauge: Predicting the Veracity of Rumors on Twitter. In ACM Trans. Knowl. Discov. Data 11, 4, Article 50 (August 2017), 36 pages. find
  23. Evan Williams, Paul Rodrigues, and Valerie Novak. 2020. Accenture at CheckThat! 2020: If you say so: Post-hoc fact-checking of claims using transformer-based models. In Cappellato et al. (2020). find
  24. Xinyi Zhou, and Reza Zafarani. 2018. A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities. In arXiv:arXiv-1812. ↗
  25. Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter. 2018. Detection and Resolution of Rumours in Social Media.In ACM Computing Surveys. 51(2): pages 1–36. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hussein-2021-damascusteam-nlp4if2021-fighting-arabic-covid/