Elamur Research⌕
☰
home / papers / 2026 / When Bigger Isn't Better: Evaluating LLMs for Arabic Sentiment Analysis

When Bigger Isn't Better: Evaluating LLMs for Arabic Sentiment Analysis

Paper · 2026 · Mohamed Ibrahim, Abdullah Makki, Youssef Barakat et al.

Summary

The paper asks whether a small transformer fine-tuned for Arabic sentiment still beats general-purpose LLMs used through zero-shot prompting. The authors compare CAMeL-MSA, a CAMeL-BERT model of about 110M parameters fine-tuned for sentiment, against eight LLMs from the Qwen, BLOOMz, AraGPT2 and Gemma families. The test bed has six corpora under one three-class protocol (positive, negative, neutral), scored with Accuracy and Macro F1:

  • three classic benchmarks (LABR, ASTD, ArSAS);
  • a small set of Saudi customer-feedback audio, transcribed automatically by their ASR system;
  • two synthetic sets generated with GPT-4.1 Nano.

The fine-tuned model wins clearly on five of the six datasets. On ASTD every system does badly. The LLMs mostly stay in a band of roughly 28–46% accuracy, close to chance for balanced three-class data. The authors also document a labeling problem in LABR that hurts reproducibility. For practitioners choosing between prompting a big model and deploying a compact specialized one, the study suggests that size alone does not buy task accuracy.

Key points

  • Models. The baseline is CAMeL-MSA (CAMeL-BERT, 110M). The eight LLMs are Qwen2.5-3B, Qwen2.5-7B-Instruct, Qwen3-8B, AraGPT2-medium, BLOOMz-1b7, BLOOMz-3b, BLOOMz-7b1 and Gemma-3-4b-it. All LLMs were run zero-shot.
  • Datasets. All were balanced through stratified sampling:
    • LABR: 7,056 reviews; stars 1–2 map to negative, 3 to neutral, 4–5 to positive.
    • ASTD: 2,415 tweets; objective and mixed labels map to neutral.
    • ArSAS: 13,200 tweets.
    • Transcribed: 97 ASR samples.
    • Saudi: 300 synthetic samples.
    • Multilingual multi-dialect: 15,000 synthetic samples, including Arabic–English mixing.
  • ArSAS. CAMeL-MSA scored 86.7% accuracy and 0.866 Macro F1. The strongest LLM, BLOOMz-3b, reached 44.1% and 0.35.
  • LABR. CAMeL-MSA scored 47% and 0.45, against 36.3% and 0.28 for Qwen2.5-3B.
  • Other sets. CAMeL-MSA reached 97.2% on Multilingual, 93.3% on Transcribed and 100% on Saudi. On ASTD it fell to 18.4%, below AraGPT2-medium at 34.5%.
  • LABR labeling issue. Some LABR raters apparently treated 1 star as the best score, so the star-to-class mapping is unreliable. The authors flag this as a reproducibility risk.
  • Caveats from the authors. The small and synthetic sets may inflate scores and reflect generator bias. LLM results may also depend on prompt and decoding choices.

Interesting ideas

  • I1. Feeding raw, uncorrected ASR transcripts into evaluation turns real transcription errors into a built-in robustness test, instead of testing only on clean text.
  • I2. The authors audited the label mapping itself and found that the direction of the star scale was inconsistent between raters, which silently corrupts gold labels.
  • I3. On balanced three-class data, many LLM scores sit near 33%, so chance level is the natural yardstick for whether a system learned anything.
  • I4. The authors treat tiny or LLM-generated sets as supplementary stress tests, explicitly not as proof of readiness, because perfect scores there may only reflect generator bias.
  • I5. The study runs one fixed protocol (same labels, metrics and balancing) across many corpora instead of reporting a single benchmark, which exposes dataset-dependent swings such as the ASTD collapse.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Add a noisy-input test set (ASR-like or OCR-like text) so the tokenizer, tagger and lemmatizer are measured on degraded input, not only on edited prose.
  • I2 (tree): When converting treebanks between UD dialects or annotation schemes, record every mapping rule and check samples for flipped or ambiguous conventions before trusting converted gold data.
  • I3 (learn): Report a chance or majority baseline next to every metric as a negative control, so near-random components are caught at once.
  • I4 (learn): Data produced by LLM teacher tiers should be scored and reported separately from human-built tests, with near-perfect results treated as a warning sign.
  • I5 (answer): Evaluate readers across several distinct test families (school tests, maths word problems, fables) under one shared scoring protocol, so a gain on one family is not mistaken for general progress.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSentiment & emotion · 1,035 worksSemitic languages · 3,066 works

Cite

Cite as Mohamed Ibrahim, Abdullah Makki, Youssef Barakat, Nour Samy and Sarah AlHumoud. 2026. When Bigger Isn't Better: Evaluating LLMs for Arabic Sentiment Analysis. The 7th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT7) with 5 Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2kimof4u6y8x

@inproceedings{ibrahim2026when,
  title = {{When Bigger Isn't Better: Evaluating LLMs for Arabic Sentiment Analysis}},
  author = {Mohamed Ibrahim, Abdullah Makki, Youssef Barakat, Nour Samy and Sarah AlHumoud},
  booktitle = {The 7th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT7) with 5 Shared Tasks},
  year = {2026},
  doi = {10.63317/2kimof4u6y8x},
  url = {https://doi.org/10.63317/2kimof4u6y8x}
}

References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alahmadi, D. (2025). Human Versus AI: A Comparative Study of Zero-Shot LLMs and Transformer Models Against Human Annotations for Arabic Sentiment Analysis. International Journal of Advanced Computer Science and Applications, 16(8). (Match pages/DOI to the published PDF.) ↗
  2. Alduailej, A., & Alothaim, A. (2022). AraXLNet: Pre-trained language model for sentiment analysis of Arabic. Journal of Big Data, 9(1), 72. https://doi.org/10.1186/s40537-022-00625-z ↗
  3. Al-Twairesh, N., Al-Khalifa, H., & Al-Salman, A. (2017). ArSAS: An Arabic Saudi sentiment analysis dataset. In Proceedings of the 24th International Conference on Neural Information Processing (ICONIP). find
  4. Aly, M., & Atiya, A. (2013). LABR: A large-scale Arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 494–498). Association for Computational Linguistics. https://aclanthology.org/P13-2088 ↗
  5. Antoun, W., Baly, F., & Hajj, H. (2020). AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 9–15). European Language Resources Association. https://aclanthology.org/2020.osact-1.2 ↗
  6. Antoun, W., Baly, F., & Hajj, H. (2021). AraGPT2: Pre-trained transformer for Arabic language generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (pp. 196–207). Association for Computational Linguistics. https://aclanthology.org/2021.wanlp-1.21 ↗
  7. Gemma Team. (2025). Gemma 3 technical report. arXiv Preprint arXiv:2503.19786. https://arxiv.org/abs/2503.19786 ↗
  8. Hasan, M. A., Das, S., Anjum, A., Alam, F., Anjum, A., Sarker, A., & Noori, S. R. H. (2024). Zero- and few-shot prompting with LLMs: A comparative study with fine-tuned models for Bangla sentiment analysis. In Proceedings of LREC-COLING 2024 (pp. 17808–17818). find
  9. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., & Habash, N. (2021). The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (pp. 92–104). Association for Computational Linguistics. https://aclanthology.org/2021.wanlp-1.10 Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Le Scao, T., Bari, M. S., Shen, S., Yong, Z.-X., Schoelkopf, H., Tang, X., Radev, D., Aji, A. F., Almubarak, K., Albanie, S., Alyafeai, Z., Webson, A., Raff, E., & Raffel, C. ↗
  10. (2022). Crosslingual generalization through multitask finetuning. arXiv Preprint arXiv:2211.01786. https://arxiv.org/abs/2211.01786 ↗
  11. Nabil, M., Aly, M., & Atiya, A. (2015). ASTD: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (pp. 2515–2519). Association for Computational Linguistics. https://aclanthology.org/D15-1299 ↗
  12. Polock, S. I. S. (2025). Sentiment analysis for low-resource languages using large language models with zero-shot, N-shot prompting, and LoRA fine-tuning [Master's thesis, SIS Polock]. https://oulurepo.oulu.fi/handle/10024/56755 ↗
  13. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., ... Qiu, Z. (2024). Qwen2.5 technical report. arXiv Preprint arXiv:2412.15115. https://arxiv.org/abs/2412.15115 ↗
  14. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., ... Qiu, Z. (2025). Qwen3 technical report. arXiv Preprint arXiv:2505.09388. https://arxiv.org/abs/2505.09388 ↗
  15. Zouidine, M., & Khalil, M. (2025). Large language models for Arabic sentiment analysis and machine translation. Engineering, Technology & Applied Science Research, 15(2), 20737–20742. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ibrahim-2026-when-bigger-isn-t-better/