📄 SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Images

Summary

This shared task addresses the automatic identification of propaganda and persuasion devices in memes. Memes carry their message through both overlaid text and the picture, so studying text alone misses part of the signal. Building on SemEval-2020 task 11, which covered news text only, the organisers created a multimodal corpus of memes. Annotators marked persuasion techniques in the text, including their exact spans, and also techniques expressed visually. The inventory grew to 22 techniques: 20 apply to both text and images, and 2 apply only to images. Three subtasks were defined:

• text-level multilabel classification,
• span-level tagging of techniques in the text,
• whole-meme classification using both text and image.

The task drew 71 registrations, 22 test-set submissions and 15 system papers. Transformer text encoders, mostly BERT, dominated. In the multimodal subtask, image encoders were usually ResNet variants. The results showed that both modalities contribute, and some teams found that modelling the interaction between text and image jointly worked better than simple early or late fusion. The task provides a benchmark for multimodal disinformation analysis.

Key points

• Subtasks: ST1 is multilabel classification of meme text over 20 techniques. ST2 adds span detection, making it a multilabel sequence tagging task. ST3 classifies the full meme (text plus image) over 22 techniques. Appeal to (Strong) Emotions and Transfer are image-only.
• Participation: 71 registered teams, 22 official test submissions, 15 system description papers.
• ST2 (F1-micro): Volta ranked first with .482, followed by HOMADOS (.407) and TeamFPAI (.397). A random span baseline scored .010.
• ST3 (F1-micro): Alpha won with .581, ahead of MinD (.566) and 1213Li (.549). The majority baseline, which always predicts Smears, reached .354, and the random baseline reached .071.
• Alpha's system was an ensemble of DeBERTa+ResNet, DeBERTa+BUTD and ERNIE-VIL. It used grid features from ResNet50 and salient-region features from BUTD.

Interesting ideas

• I1. Modality-specific labels. The label inventory explicitly separates techniques that only make sense visually from those shared by text and image. This lets one corpus support both text-only and multimodal subtasks, a useful design for any annotation scheme covering mixed media.
• I2. Copyright-safe examples. All meme examples in the paper are the authors' own recreations built from images with clear licences. This is a practical way to illustrate sensitive social-media data in publications without redistributing copyrighted content.
• I3. Span detection as QA. TeamFPAI reframed span-level technique detection as machine reading comprehension, posing a question per technique and extracting answer spans. They combined this with data augmentation and loss design to cope with sparse, imbalanced labels. The same reframing could suit other low-resource span-labelling tasks.
• I4. Label embeddings with attention. 1213Li learned an embedding for each label and used multimodal attention to score how well each label matches the fused text-image features. This approach helps when there are many labels with uneven frequencies.
• I5. Exploiting meme layout. MinD added positional embeddings derived from OCR text boxes and a face recognition component. Layout and the people depicted can be cheap but informative signals for analysing memes or other image-text content.