📄 ISHFMG_TUN at StanceEval: Ensemble Method for Arabic Stance Evaluation System

Summary

The paper presents a system for the StanceEval shared task, which asks whether an Arabic tweet is in favor of, against, or neutral toward a given target. The authors use the MAWQIF corpus, which has 4,121 tweets on three targets. Arabic is hard to process here because of its rich morphology, dialect variation and inconsistent orthography. Instead of a neural model, they use a classical pipeline. Tweets are cleaned and normalized, and SMOTE is applied to counter class imbalance. Each tweet is represented by several TF-IDF n-gram views at the word and character levels, and five scikit-learn classifiers vote on the label by hard majority. Trained on train+dev, the ensemble reaches an overall test F1 of 0.7027. Scores vary by topic: women empowerment is easiest and digital transformation hardest. The neutral "None" class is mostly missed. The work shows that a cheap, transparent ensemble gives a reasonable baseline for Arabic stance detection. The authors suggest transformer models such as MARBERT as a next step.

Key points

• Data: MAWQIF tweets are labeled for stance, sentiment and sarcasm. The three targets are COVID-19 vaccine (1,373 tweets), digital transformation (1,348) and women empowerment (1,400).

• Preprocessing: The authors strip punctuation, digits, non-Arabic tokens, repeated letters, emoticons, stop words and diacritics, then normalize the text. SMOTE then oversamples the minority classes.

• Features: Scikit-learn FeatureUnion combines three TF-IDF views:

  • word 1–5-grams;
  • character 1–4-grams;
  • character 1–5-grams that respect word boundaries.

• Classifiers: Five models vote by hard majority:

  • SGD (alpha 1e-5, l2);
  • LinearSVC (l2, tol 0.001);
  • Multinomial NB (alpha 0.01);
  • Ridge (alpha 1);
  • Random Forest (default settings).

  The design follows earlier work on Arabic dialect identification (Ghoul and Lejeune, 2020).

• Results: Overall F1 is 0.7485 on dev and 0.7027 on test.

  Target	Dev F1	Test F1
  Women empowerment	0.7927	0.7393
  COVID vaccine	0.7548	0.7019
  Digital transformation	0.6981	0.6670

  On test, "None" gets only 10 true positives against 50 false negatives. "Against" gets 118 TP, 66 FP and 62 FN.

Interesting ideas

• I1. A few diverse, cheap models (linear, probabilistic, tree-based) that settle a label by majority can be more robust than any one of them, with no tuning of the combination needed.
• I2. Character n-grams that stop at word boundaries capture affixes and stems without a morphological analyzer, and sit alongside word n-grams as a separate view.
• I3. The headline F1 hid an almost unrecognized neutral class; only per-label TP/FP/FN exposed it.
• I4. Splitting scores by target topic showed accuracy depends on the subject, which points to where more work is needed.
• I5. Synthetic oversampling of minority classes in feature space is a simple fix for skewed label distributions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Treat each witness parser like an ensemble member and accept a silver arc or label only on majority agreement, recording which witnesses dissented as an explanation.
• I2 (tokens): Boundary-aware character n-gram features could back off the tagger and lemmatizer for unknown or misspelled English words, guessing suffix-driven tags such as -ness or -ized.
• I3 (learn): Evaluation reports should list per-class confusion counts alongside negative controls, so a collapsing rare label cannot hide inside a good average.
• I4 (answer): Break reader accuracy down by test type (school course, word problem, fable QA) to find which genres the world-of-text pipeline handles poorly.
• I5 (learn): For rare frame or semantic-role labels in training data, targeted augmentation could balance the classes, but synthetic items should be kept out of evaluation sets.