Elamur Research⌕
☰
home / papers / 2023 / HTE at ArAIEval Shared Task: Integrating Content Type Information in Binary Persuasive Technique Detection

HTE at ArAIEval Shared Task: Integrating Content Type Information in Binary Persuasive Technique Detection

Paper · 2023 · Khaldi Hadjer and Taqiy Bouklouha

Summary

The authors address binary detection of persuasive techniques in Arabic text that mixes two genres: tweets and news paragraphs. Their entry to ArAIEval 2023 task 1A tests whether telling a transformer the genre of each snippet improves detection. The data is the official training set only, about 2.4k items, skewed towards news and towards persuasive labels. On top of MARBERT or AraBERT encoders they compare three designs. The first is a plain classifier. The second concatenates a genre feature onto the sentence vector. The third adds a second head that predicts the genre, trained jointly with the main head by summing the two losses. Each design is trained with either cross-entropy or focal loss, giving twelve configurations. On the dev set, every variant that used genre information beat the corresponding plain baseline. The best system, the multitask model trained with focal loss, gained about 3 points. It ranked first of 13 teams on the test set. The work shows that cheap metadata can act as an auxiliary task and help a single model handle text from different sources.

Key points

  • Training data: 1,575 news paragraphs (65%) and 852 tweets (35%). Overall, 79% of items are persuasive. News snippets average about 211 characters and tweets about 100.
  • Architecture: a shared encoder (MARBERT or AraBERT) feeds either an injected genre feature or an auxiliary genre-prediction head. The two options are mutually exclusive.
  • Best dev micro-F1 was 0.8610, from the multitask model with focal loss. The paper is inconsistent about which encoder this model used: the results section says AraBERT, while the conclusion says MARBERT.
  • Official test results: micro-F1 76.34% and macro-F1 73.21%, first of 13 participants. Code is on GitHub (TaqiyEddine-B/Transformers-for-Propaganda-Detection).
  • Error analysis: 119 of 503 test items were wrong, and 103 of those were news paragraphs. About 75% of tweet errors were non-persuasive tweets labelled persuasive, which the authors link to label skew.

Interesting ideas

  • I1. Turning free metadata such as genre into an auxiliary prediction target makes the shared encoder learn source-specific cues without any extra annotation.
  • I2. The paper compares two ways of using the same side information: feeding it in as an input feature versus asking the model to predict it. This is a clean ablation for deciding how metadata should enter a model.
  • I3. Focal loss is used to counter heavy label imbalance instead of resampling the data.
  • I4. Errors are broken down per genre with separate confusion matrices, which showed that one genre fails mostly through false positives.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could carry a cheap side head that predicts the source register (news, fable, school text), pushing its encoder to notice register-dependent syntax.
  • I2 (learn): when training on silver data from mixed sources, both variants could be run against a no-metadata control to check whether source tags really help or just leak.
  • I3 (tokens): rare morphological tags or lemma classes in the tagger could be trained with focal-style down-weighting of easy, frequent cases.
  • I4 (answer): evaluation of the readers should report errors separately for fables, word problems and school tests, and by error direction, so systematic biases become visible and explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Khaldi Hadjer and Taqiy Bouklouha. 2023. HTE at ArAIEval Shared Task: Integrating Content Type Information in Binary Persuasive Technique Detection. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.46

@inproceedings{hadjer2023hte,
  title = {{HTE at ArAIEval Shared Task: Integrating Content Type Information in Binary Persuasive Technique Detection}},
  author = {Khaldi Hadjer and Taqiy Bouklouha},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.46},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.46}
}

References

22 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 108–118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  3. Alberto Barrón-Cedeño, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing Management, 56(5):1849– 1864. ↗
  4. Buse Carik and Reyyan Yeniterzi. 2021. SU-NLP at CheckThat! 2021: Check-worthiness of Turkish tweets. find
  5. Nelson Filipe Costa, Bryce Hamilton, and Leila Kosseim. 2023. CLaC at SemEval-2023 task 3: Language potluck RoBERTa detects online persuasion techniques in a multilingual setup. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1613–1618, Toronto, Canada. Association for Computational Linguistics. persuasion techniques detection using multilingual models. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1237–1244, Toronto, Canada. Association for Computational Linguistics. find
  6. Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2019. An interactive multi-task learning network for end-to-end aspect-based sentiment analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 504–515, Florence, Italy. Association for Computational Linguistics.
  7. Yi Huang, Buse Giledereli, Abdullatif Köksal, Arzucan Özgür, and Elif Ozkirimli. 2021. Balancing methods for multi-label text classification with long-tailed class distribution. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8153–8161. find
  8. Hadjer Khaldi, Farah Benamara, Camille Pradel, and Nathalie Aussenac Gilles. 2022. A closer look to your business network: Multitask relation extraction from economic and financial french content. In The AAAI-22 Workshop on Knowledge Discovery from Unstructured Data in Financial Services. find
  9. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988. ↗
  10. Jianyi Liu, Xi Duan, Ru Zhang, Youqiang Sun, Lei Guan, and Bingjie Lin. 2021. Relation classification via bert with piecewise convolution and focal loss. Plos one, 16(9):e0257092. ↗
  11. Giovanni Da San Martino, Alberto Barrón-Cedeno, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 task 11: Detection of propaganda techniques in news articles. In Proceedings of the 14th Workshop on Semantic Evaluation, SemEval ’20, pages 1377–1414. ↗
  12. Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1–10, Vancouver, Canada. Association for Computational Linguistics.
  13. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. SemEval-2021 task 6: Detection of persuasion techniques in texts and images. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’21, pages 70–98. ↗
  14. Yili Ma, Liang Zhao, and Jie Hao. 2020. XLP at SemEval-2020 task 9: Cross-lingual models with focal loss for sentiment analysis of code-mixing language. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 975–980, Barcelona (online). International Committee for Computational Linguistics. find
  15. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abdelhakim Freihat. 2023a. Araieval shared task: Persuasion techniques and disinformation detection in arabic text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  16. Hamdy Mubarak, Samir Abdaljalil, Azza Nassar, and Firoj Alam. 2023. Detecting and reasoning of deleted tweets before they are posted. arXiv preprint arXiv:2305.04927. ↗
  17. Maram Hasanain, Ahmed El-Shangiti, Rabindra Nath Nandi, Preslav Nakov, and Firoj Alam. 2023b. QCRI at SemEval-2023 task 3: News genre, framing and Antonio Purificato and Roberto Navigli. 2023. APatt at SemEval-2023 task 3: The sapienza NLP system for ensemble-based multilingual propaganda detection. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 382– 388, Toronto, Canada. Association for Computational Linguistics. find
  18. Shabnam Tafreshi and Mona Diab. 2018. Emotion detection and classification in a multigenre corpus with joint multi-task deep learning. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2905–2913, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  20. Prashanth Vijayaraghavan and Soroush Vosoughi. 2022. TWEETSPIN: Fine-grained propaganda detection in social media using multi-view representations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3433–3448, Seattle, United States. Association for Computational Linguistics. ↗
  21. Ben Wu, Olesya Razuvayevskaya, Freddy Heppell, João A. Leite, Carolina Scarton, Kalina Bontcheva, and Xingyi Song. 2023. SheffieldVeraAI at SemEval-2023 task 3: Mono and multilingual approaches for news genre, topic and persuasion technique classification. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1995–2008, Toronto, Canada. Association for Computational Linguistics. ↗
  22. Wei Ye, Bo Li, Rui Xie, Zhonghao Sheng, Long Chen, and Shikun Zhang. 2019. Exploiting entity BIO tag embeddings and multi-task learning for relation extraction with imbalanced data. In Proceedings of ACL, pages 1351–1360. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hadjer-2023-hte-araieval-shared-task-integrating/