📄 Ensemble Language Models for Multilingual Sentiment Analysis

Summary

The paper looks at three-way tweet sentiment classification (positive, negative, neutral) in English and Arabic. Arabic is treated as the low-resource side. English data comes from SemEval-17 Task 4A plus earlier SemEval rounds. Arabic data merges SemEval-17 Arabic tweets with ASTD.

Four pretrained transformers were fine-tuned separately for each language: AraBERTv02, RoBERTa base, multilingual BERT (the baseline) and XLM-RoBERTa base. Two ensemble architectures were then built. Each joins a language-specific encoder with m-BERT by concatenating their pooled outputs, passing them through a fusion layer and then a feed-forward classifier. One variant adds multi-head attention before the classifier. A majority-voting ensemble and training on the combined bilingual data were also tested. Macro-F1 is the main metric.

Monolingual models clearly beat m-BERT, especially on Arabic. Majority voting gives the best English result. On Arabic, the plain AraBERTv02 model stays ahead of every ensemble. The study is a modest course-style project, but it gives a useful picture of how much language-specific pretraining matters compared with multilingual sharing.

Key points

• Data:
  • English train set: SemEval 2013–2016. The 2013/2014 test sets serve as dev, and the SemEval-17 test set is held out for evaluation.
  • Arabic: SemEval-17 subtasks A, B and D merged with ASTD (ASTD's "Objective" class removed), split 90/10 into train and validation. The official SemEval-17 subtask A test set is used for testing.
• Fine-tuning settings: Adam with learning rate 2e-5, max length 256, batch size 16, 3 epochs, and BCE-with-logits loss.
• English macro-F1:
  • m-BERT 67.06, RoBERTa 70.84, XLM-RoBERTa 69.13.
  • Majority voting is best at 71.03.
  • The feed-forward ensemble scores 68.59 when trained on English only and 69.88 when trained on combined data.
• Arabic macro-F1:
  • m-BERT 53.08, AraBERTv02 69.78 (best), XLM-RoBERTa 63.74.
  • Ensembles land at 66.4–67.8.
  • The multi-head attention variant trained on combined data reaches only 66.42.
• Limitations stated by the author: spot checks found mislabeled and biased annotations, no inter-annotator agreement was published, and no error analysis was done.

Interesting ideas

• I1. Feeding a model both a language-specialised encoder and a broad shared encoder, then fusing their pooled vectors, lets specific and general knowledge complement each other without retraining either from scratch.
• I2. Simple majority voting over separately fine-tuned models matched or beat the learned fusion architectures, so a voting ensemble is a cheap and strong reference that any fancier combiner should have to beat.
• I3. Before merging two corpora, the author removed a label that existed in only one scheme so the class inventory matched, which is a minimal and transparent form of harmonisation.
• I4. Earlier editions' test sets are reused as the development split, so the newest official test set stays fully untouched for final scoring.
• I5. Hand-sampling a few gold items exposed annotation mistakes and bias, a reminder that gold labels themselves need auditing.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could concatenate a small domain-tuned encoder with a general one before the arc scorer, and ablations would show which source drives each attachment decision.
• I2 (learn): when witness parsers agree on silver trees, always report plain vote-agreement accuracy next to any weighted or learned combiner as a baseline control.
• I3 (tree): during UD dialect conversion, explicitly list relations that have no counterpart in the target scheme and drop or flag them rather than mapping them silently.
• I4 (learn): rotate older evaluation sets into tuning duty and keep a freshest held-out set that no repair rule or teacher tier has ever seen.
• I5 (learn): periodically sample gold treebank and answer-key items for manual review, and log disagreements as a separate error category instead of counting them as model failures.