Elamur Research⌕
☰
home / papers / 2026 / AraSSM: A bidirectional state-space encoder for Arabic masked language modeling

AraSSM: A bidirectional state-space encoder for Arabic masked language modeling

Paper · 2026 · Ahmed Amine Aliane, Hassina Aliane and Nasredine Semmar

Summary

Today, Arabic natural language understanding is dominated by Transformer-based encoders, including AraBERT, MARBERT and CAMeLBERT. Their attention cost grows quadratically with input length. Earlier attempts to use Mamba for Arabic skipped language-specific pretraining and fell behind. The authors build AraSSM, a 12-layer bidirectional Mamba encoder. Each block runs one selective scan left-to-right and another right-to-left, then fuses the two with a learned projection, so cost stays linear in sequence length. The model is pretrained from scratch with masked language modeling on roughly 80GB of cleaned, chunk-level deduplicated Arabic Wikipedia and CulturaX text. Training used only four RTX 2080Ti cards for about ten days. The authors fine-tune it under the AraBERT per-task protocol with three seeds. It equals or beats base-sized Transformers on HARD sentiment, comes close on ARCD question answering and ANERcorp NER, and trails by a few points on XNLI-ar. The work shows that a modest academic compute budget can produce a nearly competitive Arabic encoder, and the checkpoint is public. A matched-compute Transformer baseline and long-sequence speed tests are left for future work.

Key points

  • AraSSM has 12 layers, hidden size 512 and 105.0M parameters, close in size to AraBERT-base (~110M, hidden size 768). It reuses the AraBERTv02 tokenizer so that results stay comparable with the AraBERT family.
  • All pretraining and fine-tuning ran on 4× RTX 2080Ti (11GB) for about ten days, roughly 960 GPU-hours.
  • Results (mean ± std over three seeds):
    • HARD: 96.37 ± 0.03% accuracy
    • ARCD: 32.19 ± 1.07 EM and 63.79 ± 0.25 F1
    • ANERcorp: 81.54 ± 0.30 entity-level F1
    • XNLI-ar: 72.83 ± 0.07% accuracy
  • The authors flag several benchmark caveats:
    • ANERcorp has known label noise.
    • XNLI-ar and Arabic-SQuAD (the latter used for prefinetuning) contain machine-translation artifacts.
    • ARCD has a small test set, and F1 is treated as more informative than EM on it.
  • Evaluation covers only the standard 512-token regime. The checkpoint is released as aliane29/arassm-base on the Hugging Face Hub.

Interesting ideas

  • I1. When reversing a padded batch for the backward pass, flip only the real tokens in place, and mask padding both before the backward scan and after the block, so a sentence's output never depends on what else is in the batch.
  • I2. Keep the tokenizer of the baseline fixed so that any score difference can be credited to the architecture rather than to segmentation.
  • I3. Report every number as mean ± standard deviation across several fine-tuning seeds instead of a single lucky run.
  • I4. Name each benchmark's built-in ceiling (label noise, translation artifacts, tiny test sets) and favour token-level F1 over exact match when valid answers differ only in surface form.
  • I5. A linear-time bidirectional encoder keeps whole-document context affordable where attention would force truncation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could adopt the same discipline for any right-to-left component and add a test that a sentence gets an identical parse whether run alone or inside a mixed-length batch.
  • I2 (learn): when comparing parser or tagger variants, freeze the crate en tokenization and tagging layer so that gains are attributed to the component actually changed.
  • I3 (learn): training runs for the perceptron parser and gparse could log multi-seed spreads, so small LAS gains are judged against seed noise.
  • I4 (answer): QA on fables and tales could score with partial-overlap F1 next to exact match and document known gold-label errors, so misses are not wrongly blamed on the reader.
  • I5 (frames): coreference across a full tale could use such an encoder to see all mentions at once without chunking the text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Commissariat à l'Énergie Atomique et aux Énergies AlternativesParis · France11 works · 416 citations🏛️Laboratoire d'Intégration des Systèmes et des TechnologiesPalaiseau · France8 works · 321 citations🔬Centre de Recherche sur l'Information Scientifique et TechniqueAlgiers · Algeria3 works · 32 citations🎓University of Algiers Benyoucef BenkheddaAlgiers · Algeria4 works · 4 citations🔬CEA Paris-Saclay - Etablissement de SaclaySaclay · France4 works · 207 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 works

Cite

Cite as Ahmed Amine Aliane, Hassina Aliane and Nasredine Semmar. 2026. AraSSM: A bidirectional state-space encoder for Arabic masked language modeling. arXiv:2608.08256.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2608.08256 · arXiv 2608.08256

@misc{aliane2026arassm,
  title = {{AraSSM: A bidirectional state-space encoder for Arabic masked language modeling}},
  author = {Ahmed Amine Aliane, Hassina Aliane and Nasredine Semmar},
  year = {2026},
  doi = {10.48550/arxiv.2608.08256},
  eprint = {2608.08256},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2608.08256}
}

References

22 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wissam Antoun, Fady Baly, and Hazem Hajj. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, pages 9–15, 2020. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 7085–7105, 2021. ↗
  3. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92–104, 2021.
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
  5. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arxiv preprint arxiv:2312.00752, 2023. ↗
  6. Ling Yue, Sixue Xing, Yingzhou Lu, and Tianfan Fu. BioMamba: A pre-trained biomedical language representation model leveraging mamba. arxiv preprint arxiv:2408.02600, 2024. ↗
  7. Zhichao Yang, Avijit Mitra, Sunjae Kwon, and Hong Yu. ClinicalMamba: A generative clinical language model on longitudinal clinical notes. arxiv preprint arxiv:2403.05795, 2024. ↗
  8. Ali. Al-Zawqari, Mohamed Ahmed, Abul Gabbar Al-Sharafi, Mohammad M. Khader, Ali Safa, and Gerd Vandersteen. Neural classification of argument elements and styles in Arabic competitive debates. In IEEE Access, 2025. find
  9. Wissam Antoun, Fady Baly, and Hazem Hajj. AraELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, 2021. ↗
  10. Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat, and Hussein Al-Natsheh. ALUE: Arabic language understanding evaluation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, 2021.
  11. AbdelRahim Elmadany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. ORCA: A challenging benchmark for Arabic language understanding. arxiv preprint arxiv:2212.10758, 2023. ↗
  12. Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arxiv preprint arxiv:2111.00396, 2021. ↗
  13. Lainghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision Mamba: Efficient visual representation learning with bidirectional state space model. arxiv preprint arxiv:2401.0917, 2024. ↗
  14. Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. VMamba visual state space model. arxiv preprint arxiv:2401.10166, 2024. ↗
  15. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu MAn, Nghia Thrung Ngo, Frank Dernoncourt, Ryan A. Rossi, and Nguyen Huu. CulturaX:a cleaned,enormous and multilingual dataset for large language models in 167 languages. arxi preprint arxiv:2309.09400, 2023. ↗
  16. Ashraf Elnagar, Yasmin S. Khalifa, and Anas Einea. Hotel Arabic-reviews dataset construction for sentiment analysis applications. In Intelligent Natural Language Processing: Trends and Applications, 2018. ↗
  17. Yassine Benajiba, Paolo Rosso, and José Miguel Benedí. ANERsys: An Arabic named entity recognition system based on maximum entropy. In Proceedings of CICLing, 2007. find
  18. Rami Al-Rfou, Vivek Kulkarni, Bryan Perozzi, and Steven Skiena. Polyglot-NER: Massive multilingual named entity recognition. In Proceedings of the SIAM International Conference on Data Mining, 2015. ↗
  19. Mashael Al-Duwais, Hend Al-Khalifa, and Abdulmalik Al-Salman. CleanANERCorp: Identifying and correcting incorrect labels in the anercorp dataset. In OSACT 2024 Workshop, pages 13–19, 2024. find
  20. Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, 2019.
  21. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470, 2020. ↗
  22. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP, 2018. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aliane-2026-arassm-bidirectional-state-space-encoder/