Elamur Research⌕
☰
home / papers / 2023 / Itri Amigos at ArAIEval Shared Task: Transformer vs. Compression-Based Models for Persuasion Techniques and Disinformation Detection

Itri Amigos at ArAIEval Shared Task: Transformer vs. Compression-Based Models for Persuasion Techniques and Disinformation Detection

Paper · 2023 · Jehad Oumer, Nouman Ahmed and Natalia Flechas Manrique

Summary

The paper reports one team's entry to the ArAIEval 2023 shared task on Arabic text. The task had two parts. Task 1 asked whether a tweet or news paragraph uses persuasion, and if so which propaganda techniques (1A binary, 1B multilabel). Task 2 asked whether a tweet carries disinformation, and of which kind (2A binary, 2B fine-grained). The main systems were fine-tuned AraBERT variants, trained with class weights to handle label imbalance. They beat the multilingual models mBERT and XLM-RoBERTa during development, and the submitted runs placed between 4th and 8th on the leaderboards.

As a second line of work, the authors reimplemented the gzip + kNN classifier of Jiang et al. (2023), which classifies texts by Normalized Compression Distance. They tested several tie-breaking rules, since a flawed tie-break had been blamed for inflated scores in the original paper. The compressor method stayed clearly below AraBERT but mostly beat the majority baseline. That makes it a cheap and transparent reference point, while also showing how fragile reported baseline numbers can be.

Key points

  • Data: 1A has 2427 train / 259 dev / 503 test snippets. 2A has 14,147 train tweets with heavy imbalance (11,491 no-disinfo vs 2656 disinfo). 2B labels are HS, OFF, SPAM and Rumor.
  • Some 2A/2B items had NaN text. The authors turned these into strings instead of dropping them, because scoring required every dev item in its original order.
  • Submitted test scores:
    • AraBERT-v2: 0.749 on 1A (5th place) and 0.550 on 1B (4th).
    • AraBERT-v02-Twitter, pretrained with an extra 60M multi-dialect tweets: 0.898 on 2A (8th) and 0.814 on 2B (7th).
    • Fine-tuning used 10 epochs, lr 2e-5 and batch size 16.
  • gzip + NCD + kNN (k=2, with tie-breaking by lowest-label-index, random choice or closer-neighbor; k=3 also tried), test scores against the majority baseline:
    • 1A: 0.658, exactly equal to the baseline.
    • 1B: 0.393 vs 0.360.
    • 2A: 0.801 vs 0.765.
    • 2B: 0.713 vs 0.505.
  • Preprocessing effects were mixed. Removing links and hashtags left AraBERT-Twitter roughly unchanged on Task 2 but sharply hurt v0.2 and v2 there. Preprocessing also degraded the compressor method, so it was run on raw text.

Interesting ideas

  • I1. The kNN tie-breaking rule, an easily overlooked detail, can noticeably move results; the authors reran the method under several explicit rules instead of trusting the original setup.
  • I2. A parameter-free compression-distance classifier gives a fast, fully inspectable floor that a model must clearly beat.
  • I3. Always printing the majority baseline exposed that one system (gzip on 1A) scored exactly the baseline, meaning it had learned nothing useful there.
  • I4. AraBERT needs the same normalization at fine-tuning time as at pretraining (spaces around digits and punctuation), otherwise its wordpiece vocabulary is misused.
  • I5. The authors note that judging truth from wording alone is unreliable, because a fake story worded like a true one evades detection; outside evidence is needed.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when witness parsers vote on silver data, make the tie policy explicit and report agreement under more than one policy, so gains do not come from an accidental default.
  • I2 (learn): use gzip + NCD kNN as a negative control in evaluations; a structured frame- or tree-based component that cannot beat it is learning surface regularities at best.
  • I3 (answer): report the majority-answer and trivial-heuristic scores next to every school-test and QA result, so apparent accuracy on skewed datasets is not mistaken for understanding.
  • I4 (tokens): pin one normalization routine shared by treebank training and inference in crate en, so tokenizer splits never drift between the two.
  • I5 (world): treat a claim as supported only if it fits the entities and states of the text world or the knowledge graph, never because of how persuasively it is phrased.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of the Basque CountryLeioa · Spain57 works · 16,520 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksInformation Systems · 1,730 worksSociology and Political Science · 626 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Jehad Oumer, Nouman Ahmed and Natalia Flechas Manrique. 2023. Itri Amigos at ArAIEval Shared Task: Transformer vs. Compression-Based Models for Persuasion Techniques and Disinformation Detection. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.53

@inproceedings{oumer2023itri,
  title = {{Itri Amigos at ArAIEval Shared Task: Transformer vs. Compression-Based Models for Persuasion Techniques and Disinformation Detection}},
  author = {Jehad Oumer, Nouman Ahmed and Natalia Flechas Manrique},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.53},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.53}
}

References

20 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. arXiv eprints, page arXiv:2201.06642. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 108–118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. ↗
  3. Ali Saleh Alammary. 2022. Bert models for arabic text classification: a systematic review. Applied Sciences, 12(11):5720. ↗
  4. Shatha Alyoubi, Manal Kalkatawi, and Felwa A. Abukhodair. 2023. The detection of fake news in arabic tweets using deep learning. Applied Sciences. ↗
  5. Wissam Antoun, Fady Baly, Rim Achour, A. Hussein, and Hazem M. Hajj. 2020. State of the art models for fake news detection tasks. 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT), pages 519–524. find
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. ↗
  8. Giovanni Da San Martino, Yu Seunghak, Alberto Barrón-Cedeno, Rostislav Petrov, Preslav Nakov, et al. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 5636– 5646. Association for Computational Linguistics. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  10. Ibrahim Abu El-Khair. 2016. 1.5 billion words arabic corpus. ArXiv, abs/1611.04033. ↗
  11. Ivan Habernal, Raffael Hannemann, Christian Pollak, Christopher Klamm, Patrick Pauli, and Iryna Gurevych. 2017. Argotario: Computational argumentation meets serious games. arXiv preprint arXiv:1707.06002. ↗
  12. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Alhakim Freihat. 2023. ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  13. Takeshi Iida, Jaehyun Song, José Luis Estrada, and Yuriko Takahashi. 2022. Fake news and its electoral consequences: a survey experiment on mexico. AI & SOCIETY. ↗
  14. Madinabonu Ishmuradova. 2019. Strategies for using fake news as a tool to manipulate public opinion. find
  15. Zhiying Jiang, Matthew Yang, Mikhail Tsirlin, Raphael Tang, Yiqin Dai, and Jimmy Lin. 2023. “low-resource” text classification: A parameter-free classification method with compressors. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6810–6828, Toronto, Canada. Association for Computational Linguistics. find
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  17. Ahmed Redha Mahlous and Ali Al-laith. 2021. Fake news detection in arabic tweets during the covid-19 pandemic. International Journal of Advanced Computer Science and Applications, 12. ↗
  18. Ali Bou Nassif, Ashraf Elnagar, Omar A. Elgendy, and Yaman Afadar. 2022. Arabic fake news detection based on deep contextualized embedding models. Neural Computing & Applications, 34:16019 – 16032. ↗
  19. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  20. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oumer-2023-itri-amigos-araieval-shared-task/