📄 Benchmarking Transformer-based Language Models for Arabic Sentiment and Sarcasm Detection

Summary

Many transformer language models for Arabic have recently appeared, but each was tested on its own small set of tasks, so there was little head-to-head evidence on which works best. Abu Farha and Magdy address this by benchmarking 24 Arabic-capable models, including multilingual and Arabic-specific BERT, ELECTRA, ALBERT and GPT2 variants. They fine-tune each on two classification tasks, sentiment analysis and sarcasm detection, using the ArSarcasm-v2 dataset from the Arabic sarcasm shared task. They also record fine-tuning time.

Models pre-trained only on Arabic, especially those whose corpora include dialectal social-media text, perform best. MARBERT tops both tasks. Monolingual models beat multilingual ones, and larger models generally win when trained on comparable data. AraELECTRA nearly matches much bigger models at a fraction of the cost. AraGPT2 variants trail the BERT-style encoders, which suggests causal language models suit classification poorly.

The study, summarised in a single comparison table, gives practical guidance for choosing models in Arabic NLP. It is particularly useful when compute is limited or the input text is dialectal.

Key points

• Best results: MARBERT, pre-trained on about 128GB of text of which half is tweets, scored F PN 0.724 on sentiment and F1-sarcastic 0.584 on sarcasm. QARiB (0.72) and AraBERT-large v2 (0.724) were close on sentiment.
• Multilingual models lag: mBERT scored lowest (0.625 sentiment, 0.425 sarcasm). Large multilingual models such as XLM-R-large only reach the level of smaller Arabic-specific models.
• Efficiency: The AraELECTRA discriminator has 136M parameters and fine-tunes in about 2:40 per epoch (batch size 4). It scored 0.709 and 0.578, close to AraBERT-large, which has 371M parameters and needs roughly 5:50 per epoch.
• Fine-tuning time: Time grows steadily with parameter count. Arabic ALBERT-base was fastest (1:56 per epoch) and AraGPT2-large (792M) was slowest (11:27). AraGPT2-Mega could not be fine-tuned at all on the authors' hardware because it ran out of memory.
• AraGPT2 underperforms: All AraGPT2 variants scored about 0.63–0.67 on sentiment and about 0.52 on sarcasm, below the masked-LM encoders.