📄 Mawqif-XT: An Arabic Benchmark Dataset for Cross-Target Stance Detection

Summary

Arabic has few public resources for checking whether a stance classifier still works when the discussion topic changes. The original Mawqif corpus labels tweets for stance, sentiment, and sarcasm, but its targets are fixed, so it cannot measure transfer to new topics. The authors built Mawqif-XT, a test-only extension of 996 Arabic tweets from X on three new targets. Women Driving is close in theme to the existing Women Empowerment target. E-Cars and Trimester System come from unrelated domains. Each tweet was labeled by three annotators under the original Mawqif guidelines, and a fourth annotator settled disagreements. Two protocols are defined: training on the original targets and testing on the related one, or training on two original targets and testing on the unrelated ones. Baselines cover Arabic BERT encoders, multilingual encoders, and zero-shot 70B-class LLMs. Qwen 2.5 had the best overall scores, but no model won on every target. The resource gives Arabic stance research a standard, graded test of cross-target generalization.

Key points

• Size per target: Women Driving 352 tweets (kept from a manually checked sample of 500), E-Cars 332 (from 5,006 collected), and Trimester System 312 (from 3,000 collected).
• Label sets: stance (Favor / Against / None), sentiment (Positive / Negative / Neutral), and sarcasm (Yes / No), each annotated independently. URLs and user mentions were replaced with URL and MENTION tokens.
• Overall Fleiss' Kappa reached moderate agreement on all three tasks, with values of 0.564 (stance), 0.485 (sentiment), and 0.526 (sarcasm). About two thirds of disagreements were between the neutral label and a polar one (66.12% for stance, 67.76% for sentiment).
• Protocol 1 (related target, F avg2): Qwen 2.5 (72B) scored best on Women Driving with 73.66. The strongest encoder there was AraBERT-v02 with 65.92.
• Protocol 2 (unseen targets, F avg2):
  • Qwen 2.5 scored 80.60 on E-Cars and 74.75 overall.
  • MARBERT led on Trimester System with 74.11 and was the best encoder overall with 72.21.
  • Multilingual encoders (mBERT, DistilBERT-multilingual, XLM-R-base) stayed in the mid-50s.

Interesting ideas

• I1. The test set is graded by distance from training: one target that neighbours a training topic and two that are far away, so near transfer and far transfer are scored separately rather than blended.
• I2. The authors classify the kinds of annotator disagreement and find most are neutral versus polar, rarely polar versus polar. This explains the moderate kappa better than the number alone.
• I3. Stance toward a target, emotional tone, and sarcasm are labeled independently, and the paper explicitly keeps cases where stance and sentiment point in opposite directions.
• I4. Fine-tuned small encoders and zero-shot large LLMs are scored under the same protocols, and the per-target winner changes, so no single model is best everywhere.
• I5. Before labeling, URLs and user mentions are normalized into placeholder tokens, which removes noisy, unrepeatable strings while keeping their position in the sentence.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Held-out evaluation could be split into genres close to the treebank sources and clearly distant ones, so that a drop in accuracy can be traced to domain distance rather than hidden in one average.
• I2 (learn): When the witness parsers disagree on silver data, sorting the conflicts by type (label only, attachment, both) would show which disagreements are harmless boundary cases and which signal real errors.
• I3 (frames): crate prag could store an author's position toward an entity separately from the text's polarity and the irony flag, so that a negative-sounding sarcastic sentence can still yield a supportive stance.
• I4 (learn): LLM teacher tiers could be selected per domain or per phenomenon based on measured agreement, rather than trusting one teacher globally.
• I5 (tokens): The en tokenizer could map URLs, handles, and similar spans to typed placeholder tokens with fixed tags, so the parser sees stable inputs on web-style text.