📄 Multimodal Identification of Vaccine Content Stance on Social Media

Summary

This paper reports on a shared task, held at the EEUCA 2026 workshop co-located with ACL 2026, on automatically classifying the vaccination stance of social media memes. Memes are hard to analyse because their meaning often comes from how an image interacts with its overlaid text, and they frequently use sarcasm, irony and cultural allusions. The organizers used the VaxMeme dataset, a collection of English Twitter memes manually labelled as Vaccine-critical, Neutral or Pro-vaccine. They split it into training, validation and test portions and ranked submissions by macro-averaged F1.

Submitted systems included fine-tuned transformer encoders, CLIP- and ViLT-based fusion models, ensembles and instruction-tuned LLMs such as Qwen and Llama. The top system reached a macro F1 of 0.8494. Some text-only models were surprisingly competitive, which suggests that OCR text carries much of the stance signal. The organizers argue that visual context still helps in implicit or ironic cases. The task offers a public benchmark for multimodal analysis of public-health misinformation and points to future work on multilingual data, explainability and robustness to sarcasm.

Key points

• VaxMeme contains 10,244 image-plus-text memes collected via the Twitter API between October 2020 and April 2021. Annotator agreement was high (Fleiss' κ = 0.85).
• The class totals are 3,157 Vaccine-critical, 3,104 Neutral and 3,983 Pro-vaccine, divided 80/10/10 into train, validation and test sets.
• Participation: 77 registrants and 25 teams with evaluated systems. The best macro F1 was 0.8494.
• Team wangkongqiang placed 12th (macro F1 0.8153) with a fine-tuned Qwen2-1.5B, outperforming larger LLMs in its own experiments. In another team's experiments, a text-only BERT-base-uncased (0.8102, 14th) beat the same team's CLIP ViT-B/32 + BERT fusion models.
• The authors list three main limitations: English-only data from the COVID-19 period, noisy OCR text, and the fact that leaderboard F1 does not capture robustness or fairness.