Elamur Research⌕
☰
home / papers / 2026 / Improving on State-of-the-Art Models for Sentiment Analysis on Saudi-English Code-Switching Text

Improving on State-of-the-Art Models for Sentiment Analysis on Saudi-English Code-Switching Text

Paper · 2026 · Samaher Alghamdi, Paul Rayson and Reem Alotibi

Summary

Saudi speakers increasingly mix English words and phrases into their dialectal Arabic, both online and in speech. Until this paper, no one had tested how sentiment classifiers handle this mix. The authors built SESA-CSD, a 1k-item corpus drawn from the Saudi-English part of ArE-CSTD, a synthetic GPT-4 collection released by SDAIA. Three native Saudi annotators who are also fluent in English labeled each item as positive, negative, or neutral.

A classical voting ensemble set the reference score. The authors then fine-tuned a range of multilingual and Arabic encoders and prompted four Arabic-English LLMs in zero-shot and few-shot modes. Fine-tuned small models beat the baseline by roughly 3 to 11 points, and XLM-R-Base came out on top. The LLMs mostly fell short of the classical baseline; only Fanar exceeded it in some few-shot settings.

The work gives the first benchmark for this language pair. It also suggests that a compact fine-tuned encoder can outperform much larger prompted models on dialectal mixed text.

Key points

  • Dataset construction. SESA-CSD has 800 training and 200 test items, sampled from ArE-CSTD (Alharbi et al., 2024).
    • Items with a high English-to-Arabic word ratio were dropped so that the Saudi dialect stays dominant.
    • Items written in MSA or other dialects were swapped out while keeping the class distribution.
    • Label counts are 268 positive, 276 negative and 456 neutral.
  • Corpus statistics. Sentences average 3.89 English words and an English/Arabic ratio of 0.86. The Code-Mixing Index (CMI) is 30.29 and the Multilingual Index (MI) is 0.25.
  • Baseline. A voting classifier combining logistic regression, linear SVM and Gaussian Naive Bayes reached 85.00% accuracy and 84.60 macro-F1. Removing English tokens during cleaning cost these models about 9–10 points.
  • Fine-tuned models.
    • XLM-R-Base scored best, with 95.50% accuracy and 95.53 macro-F1.
    • It was followed by XLM-R-Large (94.00), GigaBERT-v4 (92.50), SaudiBERT and AraBERTv2-Large (both 92.00).
    • CAMeLBERT-Da (81.00) and CAMeLBERT-Mix (83.00) scored below the baseline.
  • Prompted LLMs. Each setting was run three times and accuracy averaged.
    • Fanar-1-9B-Instruct peaked at 88% with 5 shots.
    • Jais-2-8B-Chat led in zero-shot at 78.5%.
    • Most models improved at 3 shots but declined at 5 and 10 shots; Fanar was the exception.

Interesting ideas

  • I1. Sampling was controlled by the share of embedded-language words, which keeps the matrix language dominant and avoids skewed mixes; this could help anywhere a test set must reflect a target mix.
  • I2. An ablation showed that deleting the English insertions harms classification, because those tokens carry much of the evaluative signal; this argues against "cleaning" minority-language material away.
  • I3. The corpus is described with standard mixing measures (CMI, MI), not just its size, which makes it comparable to other code-switched resources.
  • I4. Accuracy for prompted models was plotted against the number of examples and averaged over three runs, revealing that extra shots can hurt; more context is not assumed to be better.
  • I5. The authors filtered GPT-4-generated text for off-target varieties and admitted it may still exaggerate mixing compared with natural usage.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): stratify silver and evaluation sets by the density of foreign or unknown tokens, so scores are not silently driven by unusually mixed sentences.
  • I2 (tokens): the tokenizer and tagger should mark foreign insertions with a language feature and keep them, rather than dropping them or treating them as noise.
  • I3 (learn): report a simple mixing or foreign-token index next to accuracy for every evaluation set, so results on different inputs can be compared fairly.
  • I4 (learn): when using LLM teacher tiers, sweep the shot count and repeat runs before fixing a prompt, rather than packing in as many examples as fit.
  • I5 (learn): treat LLM-generated training text as stylistically suspect, filter it for drift, and keep a natural-text control set to detect gains that hold only on synthetic data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Lancaster UniversityLancaster · United Kingdom35 works · 1,615 citations🎓King Abdulaziz UniversityJeddah · Saudi Arabia31 works · 508 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 4Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 worksSemitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Samaher Alghamdi, Paul Rayson and Reem Alotibi. 2026. Improving on State-of-the-Art Models for Sentiment Analysis on Saudi-English Code-Switching Text. Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.abjadnlp-1.30

@inproceedings{alghamdi2026improving,
  title = {{Improving on State-of-the-Art Models for Sentiment Analysis on Saudi-English Code-Switching Text}},
  author = {Samaher Alghamdi, Paul Rayson and Reem Alotibi},
  booktitle = {Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script},
  year = {2026},
  doi = {10.18653/v1/2026.abjadnlp-1.30},
  url = {https://doi.org/10.18653/v1/2026.abjadnlp-1.30}
}

References

42 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, and 22 others. 2025. Fanar: An arabic-centric multimodal generative ai platform. Preprint, arXiv:2501.13944. ↗
  2. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. Preprint, arXiv:2102.10684. ↗
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088– 7105, Online. Association for Computational Linguistics. ↗
  4. Aseel Addawood, Alhanouf Alsuwailem, Ali Alohali, Dalal Alajaji, Mashail Alturki, Jaida Alsuhaibani, and Fawziah Aljabli. 2020. Tracking And Understanding Public Reaction During COVID-19: Saudi Arabia As A Use Case. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online. Association for Computational Linguistics. ↗
  5. Wafia Adouane, Samia Touileb, and Jean-Philippe Bernardy. 2020. Identifying sentiments in Algerian code-switched user-generated comments. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2698–2705, Marseille, France. European Language Resources Association. find
  6. Sadam Al-Azani, Nora Alturayeif, Haneen Abouelresh, and Alhanoof Alhunief. 2024. A comprehensive framework and empirical analysis for evaluating large language models in arabic dialect identification. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1–7. find
  7. Hamed Al-Rubaiee, Renxi Qiu, and Dayou Li. 2016. Identifying Mubasher software products through sentiment analysis of Arabic tweets. In 2016 International Conference on Industrial Informatics and Computer Systems (CIICS), pages 1–6, Sharjah, Dubai, United Arab Emirates. IEEE. find
  8. Sadeen Alharbi, Raghad Aloraini, Reem BinMuqbil, Ahmed Ali, Saiful Bari, Areeb Alowisheq, and Yaser Alonaizan. 2024. Leveraging llm for augmenting textual data in code-switching asr: Arabic as an example. In Proceedings of the INTERSPEECH 2024 on Synthetic Data’s Transformative Role in Foundational Speech Models (SynData4GenAI), Kos, Greece. ISCA. find
  9. Tasneem S. Almasah, Gamal A. Ebrahim, and Marwa A. Abdelaal. 2023. A code-switched arabic-english sentiment analysis approach based on deep-learning. In 2023 International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC), pages 452–457. find
  10. Latifah Almuqren and Alexandra Cristea. 2021. Ara-Cust: a Saudi Telecom Tweets corpus for sentiment analysis. PeerJ Computer Science, 7:e510. find
  11. Kais Sultan Mousa Alowidha. 2024. English-arabic code switching and identity in bilingual saudis living in saudi arabia: A comparative study between large and small cities. Educational Administration: Theory and Practice, 30(5):4713–4722. find
  12. Dhuha Alqahtani, Lama Alzahrani, Maram Bahareth, Nora Alshameri, Hend Al-Khalifa, and Luluh Aldhubayi. 2022. Customer sentiments toward saudi banks during the covid-19 pandemic. In Proceedings of the 5th International Conference on Natural Language and Speech Processing (ICNLSP 2022), pages 251–257, Trento, Italy. Association for Computational Linguistics. find
  13. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  14. Mohamed Anwar, Abdelhakim Freihat, George Ibrahim, Mostafa Awad, Abdelrahman Atef Mohamed Ali Sadallah, Gurpreet Gosal, Gokul Ramakrishnan, Biswajit Mishra, Sarath Chandran, Ahmed Frikha, Rituraj Joshi, Etienne Goffinet, Abhishek Maiti, Ali El Filali, Sarah Al Barri, Samujjwal Ghosh, Rahul Pal, Parvez Mullah, Awantika Shukla, and 41 others. 2025. Jais 2: A family of Arabic-centric open large language models. Technical report, IFM.
  15. Gilbert Badaro, Ramy Baly, Hazem Hajj, Wassim El-Hajj, Khaled Bashir Shaban, Nizar Habash, Ahmad Al-Sallab, and Ali Hamdi. 2019. A survey of opinion mining in arabic: A comprehensive system perspective covering challenges and advances in tools, resources, models, applications, and visualizations. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 18(3). find
  16. Fazlourrahman Balouchzahi and H L Shashirekha. 2021. LA-SACo: A study of learning approaches for sentiments analysis inCode-mixing texts. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, pages 109–118, Kyiv. Association for Computational Linguistics. find
  17. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan AlRashed, Faisal Abdulrahman Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Saad Amin Hassan, Dr. Majed Alrubaian, Ali Alammari, Zaki Alawami, and 7 others. 2025. ALLam: Large language models for arabic and english. In The Thirteenth International Conference on Learning Representations.
  18. Afnan Bayazed, Ola Torabah, Redha AlSulami, Dimah Alahmadi, Amal Babour, and Kawther Saeedi. 2020. SDCT: Multi-Dialects Corpus Classification for Saudi Tweets. International Journal of Advanced Computer Science and Applications, 11(11). ↗
  19. Naaima Boudad, Rdouan Faizi, and Oulad haj thami Rachid. 2023. Multilingual, monolingual and mono-dialectal transfer learning for moroccan arabic sentiment classification. Social Network Analysis and Mining, 14. find
  20. Communications,Space and Technology Commission. 2025. Cst issued the saudi internet report 2024. https://www.cst.gov.sa/ar/media-center/news/N20 25051200 [Accessed: 2025-12-25]. ↗
  21. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics. ↗
  22. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA. Curran Associates Inc. find
  23. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the arab world. Commun. ACM, 64(4):72–81. ↗
  24. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  25. Joseph Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378–382. ↗
  26. Kamel Gaanoun, Abdou Mohamed Naira, Anass Allak, and Imade Benelallam. 2025. Darijabert: a step forward in nlp for the written moroccan dialect. International Journal of Data Science and Analytics, 20:917– 929. ↗
  27. Björn Gambäck and Amitava Das. 2014. On measuring the complexity of code-mixing. In Proceedings of the 11th International Conference on Natural Language Processing, pages 1–7, Goa, India. 1st Workshop on Language Technologies for Indian Social Media. find
  28. Gualberto Guzmán, Joseph Ricard, Jacqueline Serigos, Barbara E. Bullock, and Almeida Jacqueline Toribio. 2017. Metrics for modeling code-switching across corpora. In Interspeech 2017, pages 67–71. find
  29. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024. AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics. ↗
  30. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92–104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  31. Mohamed Amine Jerbi, Hadhemi Achour, and Emna Souissi. 2019. Sentiment analysis of code-switched tunisian dialect: Exploring rnn-based techniques. In Arabic Language Processing: From Theory to Practice, volume 1108 of Communications in Computer and Information Science, pages 122–131, Cham. Springer. find
  32. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. Working Paper 43, University of Pennsylvania, Annenberg School for Communication. find
  33. Yash Kumar Lal, Vaibhav Kumar, Mrinal Dhar, Manish Shrivastava, and Philipp Koehn. 2019. De-mixing sentiment from code-mixed text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 371–377, Florence, Italy. Association for Computational Linguistics. find
  34. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. An empirical study of pre-trained transformers for Arabic information extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4727–4734, Online. Association for Computational Linguistics.
  35. Bing Liu. 2012. Sentiment Analysis and Opinion Mining. Springer International Publishing. ↗
  36. Jiaxiang Liu, Xuyi Chen, Shikun Feng, Shuohuan Wang, Xuan Ouyang, Yu Sun, Zhengjie Huang, and Weiyue Su. 2020. Kk2018 at SemEval-2020 task 9: Adversarial training for code-mixing sentiment classification. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 817–823, Barcelona (online). International Committee for Computational Linguistics. find
  37. Gagan Sharma, R Chinmay, and Raksha Sharma. 2023. Late fusion of transformers for sentiment analysis of code-switched data. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6485– 6490, Singapore. Association for Computational Linguistics. find
  38. Ahmed Sherif and Caroline Sabty. 2024. Sentiment analysis for egyptian arabic-english code-switched data using traditional neural models and advanced language models. In Speech and Computer: 26th International Conference, SPECOM 2024, Belgrade, Serbia, November 25–28, 2024, Proceedings, Part II, page 54–69, Berlin, Heidelberg. Springer-Verlag. find
  39. Sara Stefanich, Jennifer Cabrelli, Dustin Hilderman, and John Archibald. 2019. The morphophonology of intraword codeswitching: Representation and processing. Frontiers in Communication, Volume 4 - 2019. ↗
  40. Sergei Ternovykh and Anastasia Nikiforova. 2023. Recent advances in textual code-switching. In Natural Language Processing and Information Retrieval: Principles and Applications, chapter 6, pages 159–184. CRC Press, Taylor & Francis Group. ↗
  41. Vision2030. 2025. Strategies custodian of the two holy mosques scholarship program. https://www.vision 2030.gov.sa/en/explore/strategies/custod ian-of-the-two-holy-mosques-scholarship-p rogram[Accessed: 2025-12-29]. ↗
  42. Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Pan, and Lidong Bing. 2024. Sentiment analysis in the era of large language models: A reality check. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3881–3906, Mexico City, Mexico. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alghamdi-2026-improving-state-art-models-sentiment/