Elamur Research⌕
☰
home / papers / 2026 / Evaluation of Adversarial Robustness in Arabic Language Models

Evaluation of Adversarial Robustness in Arabic Language Models

Summary

The authors ask how easily strong Arabic and multilingual transformer classifiers can be fooled by small, meaning-preserving changes to Arabic input. This area has had little study, partly because Arabic morphology, diacritics and dialects make it difficult. The study runs non-targeted, black-box attacks against five models: AraBERT, MARBERT, CaMeLBERT, mBERT and XLM-T. The attacks operate at three levels. At the character level they use diacritics, visual and phonetic changes. At the word level they use MLM substitution and conjunction manipulation. At the sentence level they use paraphrasing. The attacks combine edit-based, importance-based and paraphrase-based generation. The appendix shows that the task is sentiment classification. For each attack, the authors measure how far accuracy falls, how often the attack succeeds, what share of the input gets altered, and how closely the meaning matches the original. The authors then retrain the models on adversarial examples as a defence. Paraphrasing caused the largest average damage. Diacritic insertion broke some models while changing very little text. Adversarial training made the models more robust, mostly against word- and sentence-level attacks, while character noise remained a weak spot. The work offers an early systematic robustness benchmark for a morphologically rich language.

Key points

  • Victim models were AraBERT, MARBERT, CaMeLBERT, mBERT and XLM-T, fine-tuned on a public Hugging Face dataset. The SHAP plots in the appendix use a sentiment-analysis setup.
  • Adding diacritics lowered some models' accuracy by as much as 92% with low perturbation distance. Manipulating conjunctions kept semantic similarity high and still cost up to 58% accuracy. Paraphrasing reduced performance by 76% on average.
  • Before retraining, AraBERT was the weakest model against diacritic, visual and conjunction attacks. XLM-T suffered most under MLM attacks, and MARBERT lost the most on paraphrases.
  • After adversarial training, paraphrase robustness improved by +48.40% for MARBERT, +45.30% for CaMeLBERT and +37.40% for XLM-T. AraBERT gained more than 20% each on Diacritic, Visual and Paraphrase attacks.
  • MARBERT was the most robust model overall, and AraBERT showed the largest relative improvement. Character-level noise was still not handled well after the defence.

Interesting ideas

  • I1. Swapping function words such as conjunctions changes very little surface text and keeps meaning close, yet it can strongly mislead a model, which makes it a cheap and linguistically grounded stress test. — For Mova (learn): We could generate English variants that swap or alter conjunctions and connectives, and use them as controls to check whether parses and frames stay stable when they should, or change when the logic really changes.
  • I2. Noise that readers barely notice, such as optional diacritics or look-alike characters, can collapse a classifier, so orthographic variation should be treated as its own attack class. — For Mova (tokens): We should add normalization and tests for homoglyphs, combining marks and odd Unicode before the tokenizer and tagger, and measure how tag and lemma accuracy shift under such noise.
  • I3. Robustness is reported with several measures at once: attack success, accuracy drop, perturbation rate and semantic similarity. This keeps an attack from looking strong only because it rewrote the input heavily. — For Mova (learn): Our evaluation reports could pair every accuracy drop with how much of the input changed and how close its meaning stayed, so that real fragility is separated from invalid test items.
  • I4. Paraphrasing was the most damaging attack, which suggests that models depend on surface form more than on meaning, and that paraphrase invariance is a strong test of understanding. — For Mova (answer): We could rephrase school-test questions and fable QA items, then require the same answer and the same underlying world state, flagging any reader whose output changes with wording alone.
  • I5. Importance-based generation picks the tokens the model's decision depends on most, for example through SHAP or word-importance ranking, and perturbs those first to make attacks efficient. — For Mova (tree): For the perceptron parser and gparse, we could find the words whose change most alters the tree and focus robustness tests or silver-data augmentation on those spots.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2607.25814 · arXiv 2607.25814

@misc{alajmi2026evaluation,
  title = {{Evaluation of Adversarial Robustness in Arabic Language Models}},
  author = {Alajmi, Anwar and Salman, Ayed and Ahmad, Imtiaz},
  year = {2026},
  doi = {10.48550/arxiv.2607.25814},
  eprint = {2607.25814},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2607.25814}
}

References

69 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amodei, D., Olah, C., Steinhardt, J., Christiano, P.F., Schulman, J., Mané, D.: Concrete problems in AI safety. CoRR abs/1606.06565 (2016) 1606.06565 ↗ our card
  2. Cubuk, E.D., Zoph, B., Schoenholz, S.S., Le, Q.V.: Intriguing properties of adversarial examples. arXiv preprint arXiv:1711.02846 (2017) ↗
  3. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014) ↗ our card
  4. Wang, X., Wang, H., Yang, D.: Measure and improve robustness in NLP models: A survey. CoRR abs/2112.08313 (2021) 2112.08313 ↗
  5. Alshemali, B., Kalita, J.: Improving the reliability of deep neural networks in nlp: A review. Knowledge-Based Systems 191 , 105210 (2020) https://doi.org/10. 1016/j.knosys.2019.105210 ↗
  6. Qiu, S., Liu, Q., Zhou, S., Huang, W.: Adversarial attack and defense technologies in natural language processing: A survey. Neurocomputing 492 , 278–307 (2022) https://doi.org/10.1016/j.neucom.2022.04.020 ↗
  7. Wiyatno, R.R., Xu, A., Dia, O., De Berker, A.: Adversarial examples in modern machine learning: A review. arXiv preprint arXiv:1911.05268 (2019) ↗
  8. Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., Madry, A.: Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152 (2018) ↗
  9. Yang, Y., Rashtchian, C., Zhang, H., Salakhutdinov, R., Chaudhuri, K.: Adversarial robustness through local lipschitzness. CoRR abs/2003.02460 (2020) 2003.02460 ↗
  10. Eberhard, D.M., Simons, G.F., Fennig, C.D.: Ethnologue: Languages of the World. SIL International. Online version (2025). https://www.ethnologue.com ↗
  11. Alajmi, A., Altabaa, H., Abed, S., Ahmad, I.: Arabic question generation using transformers. ACM Transactions on Asian and Low-Resource Language Information Processing 24 (3), 1–21 (2025) find
  12. Habash, N.Y.: Introduction to Arabic Natural Language Processing. Springer (2022) find
  13. Al-Sulaiti, L., Atwell, E.S.: The design of a corpus of contemporary arabic. International journal of corpus linguistics 11 (2), 135–171 (2006) find
  14. Antoun, W., Baly, F., Hajj, H.M.: Arabert: Transformer-based model for arabic language understanding. CoRR abs/2003.00104 (2020) 2003.00104 ↗ our card
  15. Abdul-Mageed, M., Elmadany, A., et al. : Arbert & marbert: Deep bidirectional transformers for arabic. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7088–7105 (2021) find our card
  16. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., Habash, N.: The interplay of variant, size, and task type in arabic pre-trained language models. In: 6th Arabic Natural Language Processing Workshop, WANLP 2021, pp. 92–104 (2021). Association for Computational Linguistics (ACL) find our card
  17. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) ↗
  18. Barbieri, F., Anke, L.E., Camacho-Collados, J.: Xlm-t: Multilingual language models in twitter for sentiment analysis and beyond. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 258–266 (2022) find our card
  19. Morris, J.X., Lifland, E., Lanchantin, J., Ji, Y., Qi, Y.: Reevaluating adversarial examples in natural language. arXiv preprint arXiv:2004.14174 (2020) ↗
  20. Qiu, S., Liu, Q., Zhou, S., Wu, C.: Review of artificial intelligence adversarial attack and defense technologies. Applied Sciences 9 (5), 909 (2019) find
  21. Chen, Y., Su, J., Wei, W.: Multi-granularity textual adversarial attack with behavior cloning. arXiv preprint arXiv:2109.04367 (2021) ↗
  22. Cer, D., Yang, Y., Kong, S.-y., Hua, N., Limtiaco, N., John, R.S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al.: Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018) ↗ our card
  23. Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982–3992. Association for Computational Linguistics, Hong Kong, China (2019). https://doi.org/10.18653/v1/D19-1410 . https://aclanthology.org/D19-1410 ↗
  24. Shreya, G., Khapra, M.M.: A survey in adversarial defences and robustness in nlp. arXiv preprint arXiv:2203.06414 (2022) ↗
  25. Eger, S., Benz, Y.: From hero to zéroe: A benchmark of low-level adversarial attacks. In: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 786–803. Association for Computational Linguistics, Suzhou, China (2020). https://aclanthology.org/2020.aacl-main.79 ↗
  26. Formento, B., Foo, C.S., Tuan, L.A., Ng, S.K.: Using punctuation as an adversarial attack on deep learning-based NLP systems: An empirical study. In: Findings of the Association for Computational Linguistics: EACL 2023, pp. 1–34. Association for Computational Linguistics, Dubrovnik, Croatia (2023). https://aclanthology.org/2023.findings-eacl.1 ↗
  27. Chai, Y., Liang, R., Samtani, S., Zhu, H., Wang, M., Liu, Y., Jiang, Y.: Additive feature attribution explainable methods to craft adversarial attacks for text classification and text regression. IEEE Transactions on Knowledge and Data Engineering, 1–14 (2023) https://doi.org/10.1109/TKDE.2023.3270581 ↗
  28. Alshemali, B., Kalita, J.: Character-level adversarial examples in arabic. In: 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 9–14 (2021). IEEE find
  29. Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30, pp. 4765–4774. Curran Associates, Inc., ??? (2017) find our card
  30. Gao, J., Lanchantin, J., Soffa, M.L., Qi, Y.: Black-box generation of adversarial text sequences to evade deep learning classifiers. In: 2018 IEEE Security and Privacy Workshops (SPW), pp. 50–56 (2018). IEEE find our card
  31. Li, J., Ji, S., Du, T., Li, B., Wang, T.: Textbugger: Generating adversarial text against real-world applications. arXiv preprint arXiv:1812.05271 (2018) ↗ our card
  32. Morris, J.X., Lifland, E., Yoo, J.Y., Grigsby, J., Jin, D., Qi, Y.: Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. arXiv preprint arXiv:2005.05909 (2020) ↗ our card
  33. Li, L., Ma, R., Guo, Q., Xue, X., Qiu, X.: BERT-ATTACK: Adversarial attack against BERT using BERT. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6193–6202. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020. emnlp-main.500 . https://aclanthology.org/2020.emnlp-main.500 ↗ our card
  34. Garg, S., Ramakrishnan, G.: BAE: BERT-based adversarial examples for text classification. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6174–6181. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020. emnlp-main.498 . https://aclanthology.org/2020.emnlp-main.498 ↗ our card
  35. Alshemali, B., Kalita, J.: Adversarial examples in arabic. In: 2019 International Conference on Computational Science and Computational Intelligence (CSCI), pp. 371–376 (2019). IEEE find
  36. Ekbal, A., et al. : Adversarial sample generation for aspect based sentiment classification. In: Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pp. 478–492 (2022) find
  37. Alajmi, A., Ahmad, I., Mohammed, A.: Evaluating the adversarial robustness of arabic spam classifiers. Neural Computing and Applications 37 (6), 4323–4343 (2025) find
  38. Alshalan, H., Rekabdar, B.: Attacking a transformer-based models for arabic language as low resources language (lrl) using word-substitution methods. In: 2023 Fifth International Conference on Transdisciplinary AI (TransAI), pp. 95–101 (2023). IEEE find
  39. Yoo, J.Y., Qi, Y.: Towards improving adversarial training of nlp models. arXiv preprint arXiv:2109.00544 (2021) ↗
  40. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR abs/1910.01108 (2019) 1910.01108 ↗
  41. Shen, L., Zhang, X., Ji, S., Pu, Y., Ge, C., Yang, X., Feng, Y.: Textdefense: Adversarial text detection based on word importance entropy. arXiv preprint arXiv:2302.05892 (2023) ↗
  42. Huber, L., Kühn, M.A., Mosca, E., Groh, G.: Detecting word-level adversarial text attacks via shapley additive explanations. In: Proceedings of the 7th Workshop on Representation Learning for NLP, pp. 156–166 (2022) find
  43. Mozes, M., Stenetorp, P., Kleinberg, B., Griffin, L.D.: Frequency-guided word substitutions for detecting textual adversarial examples. arXiv preprint arXiv:2004.05887 (2020) ↗
  44. Zhou, Y., Jiang, J.-Y., Chang, K.-W., Wang, W.: Learning to discriminate perturbations for blocking adversarial attacks in text classification. arXiv preprint arXiv:1909.03084 (2019) ↗
  45. Zhou, Y., Zheng, X., Hsieh, C.-J., Chang, K.-w., Huang, X.: Defense against adversarial attacks in nlp via dirichlet neighborhood ensemble. arXiv preprint arXiv:2006.11627 (2020) ↗
  46. Alshahrani, N., Alshahrani, S., Wali, E., Matthews, J.: Arabic synonym bert-based adversarial examples for text classification. arXiv preprint arXiv:2402.03477 (2024) ↗
  47. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9 (8), 1735–1780 (1997) find
  48. Kim, Y.: Convolutional neural networks for sentence classification. In: Moschitti, A., Pang, B., Daelemans, W. (eds.) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746– 1751. Association for Computational Linguistics, Doha, Qatar (2014). https://doi.org/10.3115/v1/D14-1181 . https://aclanthology.org/D14-1181 ↗ our card
  49. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017) find
  50. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018) 1810.04805 ↗
  51. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) ↗
  52. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019) ↗
  53. Ribeiro, M.T., Singh, S., Guestrin, C.: ” why should i trust you?” explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016) find our card
  54. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, É., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsupervised cross-lingual representation learning at scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440–8451 (2020) find
  55. Alajmi, A.M.: Arabic Sentiment Analysis [dataset], 2023. https://huggingface.co/datasets/Anwaarma/MySentimentAnwarBig/viewer/default/train?p=1 ↗
  56. Elmadany, A., Mubarak, H., Magdy, W.: Arsas: An arabic speech-act and sentiment corpus of tweets. OSACT 3 , 20 (2018) find
  57. Barbieri, F., Espinosa Anke, L., Camacho-Collados, J.: XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 258–266. European Language Resources Association, Marseille, France (2022). https://aclanthology.org/2022.lrec-1.27 ↗ our card
  58. Abdelali, A., Darwish, K., Durrani, N., Mubarak, H.: Farasa: A fast and furious segmenter for Arabic. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pp. 11–16. Association for Computational Linguistics, San Diego, California (2016). https://doi.org/10.18653/v1/N16-3003 . https://aclanthology.org/N16-3003 ↗ our card
  59. Kudo, T., Richardson, J.: SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66–71. Association for Computational Linguistics, Brussels, Belgium (2018). https://doi.org/10.18653/v1/D18-2012 . https://aclanthology.org/D18-2012 ↗ our card
  60. Alajmi, A., Ahmad, I., Mohammed, A.: Evaluating the adversarial robustness of arabic spam classifiers. Neural Computing and Applications, 1–21 (2024) find
  61. Bani-Almarjeh, M., Kurdy, M.-B.: Arabic abstractive text summarization using rnn-based and transformer-based architectures. Information Processing & Management 60 (2), 103227 (2023) find
  62. Levenshtein, V.I., et al. : Binary codes capable of correcting deletions, insertions, and reversals. In: Soviet Physics Doklady, vol. 10, pp. 707–710 (1966). Soviet Union find
  63. Yuan, X., He, P., Zhu, Q., Li, X.: Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems 30 (9), 2805–2824 (2019) find
  64. Kadaoui, K., Magdy, S., Waheed, A., Khondaker, M.T.I., El-Shangiti, A., Abdul-Mageed, M., et al. : Tarjamat: Evaluation of bard and chatgpt on machine translation of ten arabic varieties. In: Proceedings of ArabicNLP 2023, pp. 52–75 (2023) find our card
  65. Khondaker, M.T.I., Waheed, A., Nagoudi, E.M.B., Abdul-Mageed, M.: Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp. arXiv preprint arXiv:2305.14976 (2023) ↗
  66. Bountakas, P., Zarras, A., Lekidis, A., Xenakis, C.: Defense strategies for adversarial machine learning: A survey. Computer Science Review 49 , 100573 (2023) find
  67. Zhang, C., Zhou, X., Wan, Y., Zheng, X., Chang, K.-W., Hsieh, C.-J.: Improving the adversarial robustness of nlp models by information bottleneck. arXiv preprint arXiv:2206.05511 (2022) ↗
  68. Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelovic, R., Mann, T., Kohli, P.: On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12715 (2018) ↗
  69. Goyal, S., Doddapaneni, S., Khapra, M.M., Ravindran, B.: A survey of adversarial defenses and robustness in nlp. ACM Computing Surveys 55 (14s), 1–39 (2023) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alajmi-2026-evaluation-adversarial-robustness-arabic-language/