Elamur Research⌕
☰
home / papers / 2022 / TUB at WANLP22 Shared Task: Using Semantic Similarity for Propaganda Detection in Arabic

TUB at WANLP22 Shared Task: Using Semantic Similarity for Propaganda Detection in Arabic

Paper · 2022 · Salar Mohtaj and Sebastian Möller

Summary

The paper describes the TUB team's entry for sub-task 1 of the WANLP 2022 shared task on Arabic propaganda detection. In that sub-task, each tweet must be tagged with one or more of 21 propaganda techniques. The task is a multi-label classification problem, but the authors did not train a classifier. They treated it as semantic textual similarity (STS) retrieval instead.

Each tweet is encoded using the Arabic-language BERT released by Safaya et al. (2020). Word vectors are averaged into a single 768-dimensional sentence vector. For each test tweet, the system ranks the labelled training tweets by cosine similarity and keeps only close neighbours above a similarity cutoff. It then assigns the techniques that occur most often among those neighbours. The authors chose this design partly because Arabic lacks lexicons tied to specific techniques.

The best setting reached a micro-F1 of 0.494 on the test set, while macro-F1 was only 0.076. This shows that label transfer from neighbours can compete on frequent techniques even with a tiny training set, but it largely fails on rare ones.

Key points

  • Data: 504 training tweets and 323 test tweets, with JSON files for train, development, dev_test and test. The most frequent technique is "Loaded Language", followed by "Name calling/Labeling".
  • Preprocessing: Python re replaces Twitter handles with "username" and URLs with "weblink". URLs were deliberately not expanded into page content so that input lengths stay stable.
  • The method has three hyper-parameters:
    • n, the number of candidate neighbours;
    • a minimum cosine-similarity threshold;
    • t, the number of most frequent techniques taken from the candidates.
  • The best development setting was n=5, threshold 0.4, t=3, with micro-F1 0.497 and macro-F1 0.065. Raising the threshold to 0.6 dropped micro-F1 to 0.050.
  • Test results: micro-F1 0.494 and macro-F1 0.076. The authors suggest that combining several BERT layers, or trying other pretrained models, might help.

Interesting ideas

  • I1. Instead of learning a classifier, the system labels a new item by voting over its closest labelled examples, so every prediction comes with the training examples that caused it. This is useful when data is scarce and decisions must be justified.
  • I2. A similarity cutoff decides whether there is enough evidence to vote at all, and results proved very sensitive to it (0.4 worked, 0.6 collapsed). This shows that abstention thresholds must be tuned rather than assumed.
  • I3. Reporting both micro-F1 and macro-F1 revealed a large gap (about 0.49 vs 0.08), showing that the system mostly gets frequent labels right.
  • I4. Usernames and links are replaced by fixed placeholder tokens rather than expanded, which removes noise while keeping text length roughly unchanged.
  • I5. The authors note that some techniques depend on specific words, such as emotional words for loaded language. They used similarity only because Arabic lacked lexicons for these words.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Speech-act or irony labels in prag could be proposed from the nearest annotated frames, with those neighbours cited as the explanation.
  • I2 (learn): Thresholds for agreement among silver-data witnesses should be swept on held-out data and logged, because a strict cutoff can quietly remove nearly all output.
  • I3 (learn): Evaluation reports should place per-class or macro scores next to aggregate ones, so that rare frame or role types failing behind a strong average become visible.
  • I4 (tokens): The tokenizer could map handles, URLs and similar non-linguistic spans to typed placeholder tokens, so that tagging and parsing see stable, predictable input.
  • I5 (world): For English, explicit lexicons linked to WordNet or FrameNet could flag emotionally loaded predicates as an explainable feature, instead of relying on averaged embeddings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany77 works · 29,479 citations🎓Technische Universität BerlinBerlin · Germany22 works · 957 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Salar Mohtaj and Sebastian Möller. 2022. TUB at WANLP22 Shared Task: Using Semantic Similarity for Propaganda Detection in Arabic. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.57

@inproceedings{mohtaj2022tub,
  title = {{TUB at WANLP22 Shared Task: Using Semantic Similarity for Propaganda Detection in Arabic}},
  author = {Salar Mohtaj and Sebastian Möller},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.57},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.57}
}

References

17 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Preslav Nakov, and Giovanni Da San Martino. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop, Abu Dhabi, UAE. Association for Computational Linguistics.
  2. Habibollah Asghari, Omid Fatemi, Salar Mohtaj, Heshaam Faili, and Paolo Rosso. 2019. On the use of word embedding for cross language plagiarism detection. Intell. Data Anal., 23(3):661–680. ↗
  3. Alberto Barrón-Cedeño, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Inf. Process. Manag., 56(5):1849–1864. ↗
  4. Giovanni Da San Martino, Alberto Barrón-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 task 11: Detection of propaganda techniques in news articles. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1377–1414, Barcelona (online). International Committee for Computational Linguistics. ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.
  6. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. Detecting propaganda techniques in memes. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6603–6617, Online. Association for Computational Linguistics. ↗
  7. Prakhar Gupta and Martin Jaggi. 2021. Obtaining better static word embeddings using contextual embedding models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5241–5253. Association for Computational Linguistics. ↗
  8. Kyle Hamilton. 2021. Towards an ontology for propaganda detection in news articles. In The Semantic Web: ESWC 2021 Satellite Events, pages 230–241, Cham. Springer International Publishing. find
  9. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society. - 25th International Conference, TSD 2022, Brno, Czech Republic, September 6-9, 2022, Proceedings, volume 13502 of Lecture Notes in Computer Science, pages 251–262. Springer. ↗
  10. Salar Mohtaj, Vinicius Woloszyn, and Sebastian Möller. 2020. TUB at HASOC 2020: Character based LSTM for hate speech detection in indo-european languages. In Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, 2020, volume 2826 of CEUR Workshop Proceedings, pages 298–303. CEUR-WS.org. find
  11. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.
  12. George-Alexandru Vlad, Mircea-Adrian Tanase, Cristian Onose, and Dumitru-Clementin Cercel. 2019. Sentence-level propaganda detection in news articles with transfer learning and BERT-BiLSTM-capsule model. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 148–154, Hong Kong, China. Association for Computational Linguistics. find
  13. Vorakit Vorakitphan, Elena Cabrio, and Serena Villata. 2021. PROTECT - A pipeline for propaganda detection and classification. In Proceedings of the Eighth Italian Conference on Computational Linguistics, CLiC-it 2021, Milan, Italy, January 26-28, 2022, volume 3033 of CEUR Workshop Proceedings. CEUR-WS.org. find
  14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  15. Giovanni Da San Martino, Stefano Cresci, Alberto Barrón-Cedeño, Seunghak Yu, Roberto Di Pietro, and Preslav Nakov. 2020. A survey on computational propaganda detection. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 4826–4832. ijcai.org. find
  16. Salar Mohtaj and Sebastian Möller. 2022a. The impact of pre-processing on the performance of automated fake news detection. In Experimental IR Meets Multilinguality, Multimodality, and Interaction - 13th International Conference of the CLEF Association, CLEF 2022, Bologna, Italy, September 5-8, 2022, Proceedings, volume 13390 of Lecture Notes in Computer Science, pages 93–102. Springer. find
  17. Salar Mohtaj and Sebastian Möller. 2022b. On the importance of word embedding in automated harmful information detection. In Text, Speech, and Dialogue find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohtaj-2022-tub-wanlp22-shared-task-semantic/