📄 AHA-Memes: A Fine-Grained Multimodal Benchmark for Understanding Hate in Arabic Memes

Summary

Hateful memes express hostility through the combination of picture, caption and shared cultural background. Arabic has had no dedicated resource for studying this at a fine level, since earlier Arabic meme collections focused on propaganda or broad harm labels. The authors build AHA-Memes from four social platforms. Their pipeline removes near-duplicate images using embedding distance, keeps only memes whose text OCR can detect, and uses a mid-sized VLM to pre-screen the pool so hateful items are more common. Native Arabic speakers then label the selected memes using bilingual guidelines. Each meme gets a binary hate decision and multi-label hate types. A further pool of roughly 66K memes carries LLM-generated silver labels intended for weak supervision, not evaluation. The benchmark covers text encoders, image encoders, late fusion, and open- and closed-weight VLMs under zero-shot, retrieval-based few-shot and fine-tuning settings. A fine-tuned open VLM does best on binary detection. Embedded text carries most of the signal and images add to it. Fine-grained typing remains difficult, and zero-shot models miss many hateful memes. The resource opens culturally grounded multimodal moderation research for Arabic.

Key points

• The gold set has 5K memes (1,850 hateful, 3,150 non-hateful). Annotation was done by three trained native-speaker annotators through a vendor and cost about $4K. The taxonomy has 8 hate types and 3 non-hate types.
• Sources are Facebook, Instagram, Pinterest (manually chosen public pages) and Twitter (X) (keyword search). Near-duplicates were images with Euclidean embedding distance ≤ 3.6. Text came from EasyOCR, and memes without text were dropped.
• To pick memes for annotation, Gemma-3-12B weakly pre-labeled about 71K memes. Silver labels for the ~66K auxiliary corpus came from Gemini-3.1-Pro and cover hate, hate type, target, dialect, emotion and sentiment.
• Fine-tuned Qwen3-VL-8B reaches 0.768 binary macro-F1 and 0.334 hate-type macro-F1. Its zero-shot scores are 0.643 and 0.176. Arabic-centric text encoders beat mBERT and XLM-R-base on the binary task.
• Few-shot ICL with RRF retrieval at K=5 gives 0.708 binary and 0.241 hate-type macro-F1, and raises Hateful recall from 0.318 to 0.454. A single demonstration (K=1) performs worse than zero-shot on binary.

Interesting ideas

• I1. Few-shot demonstrations are chosen by merging two separate similarity rankings, one from text and one from image, with reciprocal rank fusion; this beats either ranking alone and costs no training. — For Mova (learn): When building prompts for LLM teacher tiers, retrieve example sentences by fusing ranks from lexical similarity and UD-tree-shape similarity, so the teacher sees cases that resemble the query both in wording and in structure.
• I2. A cheap model screens a large unlabeled pool so the rare class is better represented before paying for human labels, while negatives stay in the sample. — For Mova (learn): Use disagreement among the witness parsers to pick constructions that are rare or contested, and send those to human or top-tier review instead of sampling uniformly.
• I3. Gold data is kept strictly for evaluation, and the much larger LLM-labeled pool is explicitly limited to weak supervision. — For Mova (learn): Keep a hard boundary between silver treebank data used for training and a human-checked test set, so self-induced repair rules are never scored on data they helped create.
• I4. Errors are broken down by input text length, which shows difficulty rising with the amount of context and inference required, not only with lexical cues. — For Mova (answer): Report reader and QA accuracy bucketed by passage and sentence length to tell which stages fail once multi-step pragmatic inference is needed.
• I5. The hardest failures came from models that understood the content but read implicit attacks on a group as harmless humor, so the real difficulty is a normative boundary rather than perception. — For Mova (frames): The irony and implicature module should explicitly fill a target slot (who is mocked, and whether it is a group) separately from the humor flag, so that a joke and an attack can coexist in the frame.