📄 PICT at StanceEval2024: Stance Detection in Arabic using Ensemble of Large Language Models

Summary

The paper tackles three-way stance classification (Favor, Against, None) of Arabic tweets for the StanceEval 2024 shared task. The tweets cover three topics: COVID-19 Vaccine, Digital Transformation and Women Empowerment. The data is the MAWQIF dataset, where each tweet also carries sarcasm and sentiment labels.

The authors fine-tuned Arabic-specific BERT-style encoders (MARBERT, MARBERTv2, AraBERT, CamelBERT, QARiB and variants) in two ways:

• Single-task: stance only.
• Multi-task: stance as the main objective, with sarcasm and sentiment as auxiliary heads trained under a weighted combined loss.

Preprocessing removed links, mentions and English words, and replaced emojis with their Arabic descriptions. The authors then searched every combination of their strongest single-task and multi-task models and merged the chosen ones by majority vote.

The final ensemble reached a macro F1 of 78.02% on the blind test set, placing 7th. This shows that combining small dialect-aware encoders with auxiliary tasks is a practical recipe for low-data Arabic opinion mining. The "None" class remained the hardest category.

Key points

• Data: MAWQIF has 3502 labelled tweets, split 80:20 into 2805 training and 697 validation examples, plus a 619-tweet test set. Labels are skewed: Favor has 2148 examples and None only 333.
• Multi-task loss: stance carries four times the weight of each auxiliary task, using (sentiment + 4·stance + sarcasm) / 6. Multi-task models generally beat single-task ones on validation but sometimes overfit more.
• Experiments: 13 backbone models were tested via HuggingFace. Weighted random sampling against class imbalance gave no gain, so it was dropped.
• Final ensemble: multi-task MARBERT, single-task AraBERT and single-task CamelBERT, combined by hard voting. Ties go to the model with the best validation F1.
• Test results: 78.02% overall macro F1. By topic, Women Empowerment scored 75.52%, COVID-19 Vaccine 79.85% and Digital Transformation 78.69%. The official metric averages only the Favor and Against classes.

Interesting ideas

• I1. Train correlated side labels (sarcasm, sentiment) as extra heads, but weight the main objective several times higher so the side tasks shape the shared encoder without dominating it. — For Mova (tree): gparse could share its encoder with auxiliary heads such as UPOS or morphological features while keeping arc/label loss dominant, which may regularize the parser on small treebanks.
• I2. When an ensemble splits without a majority, the system falls back to the single member with the best validation score instead of choosing at random. — For Mova (learn): when witness parsers disagree on a silver arc, the most reliable witness could serve as the tiebreaker, and the token could be flagged as low-agreement rather than silently dropped.
• I3. Ensemble membership was picked by testing every subset of top candidates on validation data, and the winning mix included both single-task and multi-task models. — For Mova (learn): an exhaustive subset search over witness parsers could reveal which combination yields the cleanest silver data, measured against a held-out gold set.
• I4. Emojis were rewritten as textual descriptions so a word-based model can read their meaning instead of treating them as unknown symbols. — For Mova (tokens): the tokenizer could map emojis and similar symbols to lemma-like descriptions, giving the tagger and later stages an interpretable word to work with.
• I5. The neutral "None" class failed most often, which the authors attribute to its lack of distinctive wording, unlike clear support or opposition. — For Mova (frames): stance or speech-act modules should treat "no stance" as an absence of explicit evaluative frames rather than a lexical pattern, and evaluation should report it separately.