Elamur Research⌕
☰
home / papers / 2022 / AraProp at WANLP 2022 Shared Task: Leveraging Pre-Trained Language Models for Arabic Propaganda Detection

AraProp at WANLP 2022 Shared Task: Leveraging Pre-Trained Language Models for Arabic Propaganda Detection

Paper · 2022 · Gaurav Singh

Summary

The paper describes an entry in the WANLP 2022 shared task on spotting propaganda in Arabic tweets. The author took part only in Sub-task 1, where each tweet can carry several persuasion techniques at once, so the problem is multi-label classification. The method fine-tunes eight transformer encoders on the organizers' small dataset, with no extra data, no added pre-training and no preprocessing. Two of the models are multilingual: mBERT and XLM-RoBERTa. Six are Arabic-specific: bert-base-arabic, AraBERTv1, AraBERTv2, ARBERT, MARBERT and MARBERTv2. Each model gets the same light head: averaged token embeddings, dropout, a linear projection and an independent sigmoid per label. MARBERTv2, which was pre-trained largely on tweets, gave the best micro-F1 (0.611), the official metric, and its submission placed 4th. Macro-F1 stayed very low for every model, which shows that rare techniques are barely learned. The work serves mainly as a simple, reproducible baseline and shows that matching the pre-training domain to the target genre pays off.

Key points

  • Data is tiny: 504 training, 52 development and 323 test tweets. Only 18 of the 21 listed techniques actually appear in training. Loaded Language is the most frequent label at 32.8%, and about five labels account for roughly 80% of all annotations.
  • The architecture is encoder → attention-mask-aware mean pooling → StableDropout (0.3) → linear layer → sigmoid, trained with binary cross-entropy.
  • Training settings:
    • AdamW, learning rate 5e-5, weight decay 1e-2
    • batch size 4, max length 64, 10 epochs
    • run on a Colab Tesla T4
  • Post-hoc test scores:
    • MARBERTv2 reached micro-F1 0.61116 and macro-F1 0.08175.
    • bert-base-arabic had the best macro-F1 at 0.16182.
    • The submitted MARBERTv2 run scored 0.600 micro-F1 and 0.105 macro-F1 (rank 4).
  • Across models, the spread of macro-F1 (about 0.095) is roughly three times the spread of micro-F1 (about 0.039). The author attributes this gap to label imbalance.

Interesting ideas

  • I1. Comparing the spread of macro-F1 against micro-F1 across many models works as a cheap check of whether rare classes are being learned at all, or whether the score rests on frequent labels alone.
  • I2. Treating each label as its own yes/no decision, via per-label sigmoids and binary cross-entropy, lets one text carry several co-occurring tags without forcing a single choice.
  • I3. Averaging token vectors while respecting the attention mask yields a fixed-size sentence vector that ignores padding, a small detail that keeps batched pooling correct.
  • I4. The model whose pre-training text matched the target genre (tweets, in MARBERT/MARBERTv2) beat larger general or multilingual models, so genre fit can matter more than size.
  • I5. Auditing which labels actually occur in the training split (here 18 of 21) before modelling exposes classes the system can never learn.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation reports for role labelling or speech-act tagging could always show both averages and their cross-run spread, flagging label sets where the macro score is noisy and near zero.
  • I2 (frames): Speech acts, irony and implicature markers in crate prag are not mutually exclusive, so a per-label independent scorer suits them better than one softmax over all tags.
  • I3 (tree): If gparse's encoder is reused for sentence-level decisions such as clause type, mask-aware pooling avoids padding leaking into the representation.
  • I4 (learn): When choosing witness parsers or teacher models for silver data on fables and school texts, prefer those trained on similar prose, and track agreement per genre.
  • I5 (learn): Before training role or frame classifiers, check which VerbNet roles or frame labels are absent or very rare in silver data, and route them to rules or teacher tiers instead.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Gaurav Singh. 2022. AraProp at WANLP 2022 Shared Task: Leveraging Pre-Trained Language Models for Arabic Propaganda Detection. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.56

@inproceedings{singh2022araprop,
  title = {{AraProp at WANLP 2022 Shared Task: Leveraging Pre-Trained Language Models for Arabic Propaganda Detection}},
  author = {Gaurav Singh},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.56},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.56}
}

References

19 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. Arbert amp; marbert: Deep bidirectional transformers for arabic. find
  2. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Preslav Nakov, and Giovanni Da San Martino. 2022. find
  3. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop, Abu Dhabi, UAE. Association for Computational Linguistics.
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  5. European Commission, Content Directorate-General for Communications Networks, and Technology. 2018. Fake news and disinformation online. Publications Office of the European Union. find
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. ↗
  7. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5636–5646, Hong Kong, China. Association for Computational Linguistics. find
  8. Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate Arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1070–1074, Portorož, Slovenia. European Language Resources Association (ELRA).
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  10. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021a. Detecting propaganda techniques in memes. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6603–6617, Online. Association for Computational Linguistics. ↗
  11. Ibrahim Abu El-Khair. 2016. 1.5 billion words arabic corpus. ArXiv, abs/1611.04033. ↗
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
  13. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. ↗
  14. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703– 1714, Online. Association for Computational Linguistics.
  15. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931–2937, Copenhagen, Denmark. Association for Computational Linguistics.
  16. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
  18. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182, Florence, Italy. Association for Computational Linguistics. ↗
  19. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021b. SemEval-2021 task 6: Detection of persuasion techniques in texts and images. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 70–98, Online. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2022-araprop-wanlp-2022-shared-task/