📄 LomonosovMSU at SemEval-2024 Task 4: Comparing LLMs and embedder models to identifying propaganda techniques in the content of memes in English for subtasks No1, No2a, and No2b

Summary

The LomonosovMSU team took part in SemEval-2024 Task 4, which asks systems to find rhetorical and psychological persuasion techniques in English memes. Subtask 1 uses only the meme text and is hierarchical multilabel classification. Subtask 2a adds the image, and subtask 2b reduces the problem to a yes/no decision on whether any technique is present.

For subtask 1 the authors compared two approaches:

• Generative: LLaMA-2-7b-chat and Mistral-7b-Instruct-v0.2, tuned with full SFT or LoRA to output a JSON list of techniques.
• Classifier: a small trainable head on top of frozen deberta-base-cased.

For the multimodal subtasks they joined frozen CLIP or BLIP text and image embeddings and trained linear layers on the result. Labels were expanded to include every ancestor node in the technique hierarchy. Training data was the organizers' set plus filtered SemEval-2023 Task 3 samples whose labels match the current label set.

The SFT-tuned LLaMA-2 and the much cheaper frozen DeBERTa classifier scored almost the same. The team placed 18th, 9th and 11th on subtasks 1, 2a and 2b. The work suggests that lightweight classifiers on frozen embeddings remain a strong baseline against fine-tuned 7B LLMs.

Key points

• Subtask 1, development results: LLaMA-2 with SFT reached 0.65 F1, Mistral reached 0.56 F1, and the frozen deberta-base-cased classifier reached 0.638 F1.
• Subtask 1, final: the submitted LLaMA-2-7b-chat model, retrained with gold data added, scored 0.61339 F1 and placed 18th.
• LoRA failed in practice: LoRA-tuned models could not be made to return consistent JSON, and the team had no time to write rules for repairing the output format.
• Subtask 2a: CLIP reached 0.648 F1 and BLIP 0.633 F1; the team placed 9th.
• Subtask 2b: BLIP scored higher on development (0.748 vs 0.72 F1), but the final CLIP submission scored 0.772 F1 and placed 11th.
• Compute: LLM training used 4 A100 GPUs with at most 12 hours per run and vLLM for inference. CLIP and BLIP heads were trained on a single Kaggle P100.

Interesting ideas

• I1. Each gold technique was replaced by its full ancestor set (for example, Whataboutism also adds Distraction, Ad Hominem, Ethos and so on), so the model learns the coarse categories along with the leaf labels. This makes partial credit possible and supports sparse leaf classes.
• I2. The classification head runs several dropout layers in parallel on one embedding and averages their outputs before the linear layers. This is a cheap way to regularize a small head trained over a frozen encoder.
• I3. Extra training data came from an earlier shared task, but only samples whose labels exist in the current label set were kept. This avoids teaching the model categories it will never be scored on.
• I4. LoRA-tuned generators often broke the required JSON structure, which made their predictions unusable without post-hoc formatting rules. Structured LLM output needs a validation or repair step.
• I5. A frozen encoder with a small trained head nearly matched a fully fine-tuned 7B LLM at a fraction of the compute.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): When training any classifier over the WordNet/FrameNet ontology or verb classes, targets could be closed upward over hypernyms, so a wrong leaf still lands on the right branch and errors become explainable by tree depth.
• I2 (tree): gparse's arc and label heads on frozen transformer features could use this averaged multi-dropout trick to stabilize training on small treebanks.
• I3 (learn): When silver data from other corpora or UD dialects is merged, items with labels outside the target scheme could be filtered out first instead of being force-mapped.
• I4 (learn): LLM teacher tiers should check every response against a schema and reject or repair malformed answers before they enter silver data.
• I5 (frames): Speech-act or irony detectors in prag could start as light heads on frozen embeddings over explainable features, with heavy LLM tuning kept only as a teacher.