Elamur Research⌕
☰
home / papers / 2026 / NeoAraBERT: A Modern Foundation Model for Arabic Embeddings with Diacritics-Aware Tokenization and POS-Targeted Masking

NeoAraBERT: A Modern Foundation Model for Arabic Embeddings with Diacritics-Aware Tokenization and POS-Targeted Masking

Paper · 2026 · Chadi Abou Chakra, Hadi Khaled Hamoud, Osama Rakan Al Mraikhat et al.

Summary

Arabic text encoders such as AraBERT, ARBERT and MARBERT still use older BERT-era designs. They lack rotary positions, SwiGLU and Pre-RMSNorm, and they largely ignore diacritics. The authors port the NeoBERT architecture to Arabic and pre-train it on a deduplicated corpus that mixes MSA, Classical Arabic and dialects. Sources include news archives, books, OpenITI, UN transcripts, school curricula, OSCAR and Reddit. Three variants are released: MSA, dialectal (DA) and mixed (Mix).

Design choices come from Arabic-specific ablations:

  • normalization and CAMeL-based light stemming;
  • a tokenizer that writes each word bare and then appends its diacritic string;
  • masking of whole words and POS-coherent segment groups, with nouns and verbs favoured;
  • a cosine-decay learning-rate schedule.

The authors also introduce Muradif, a synonym benchmark that probes raw embeddings without any fine-tuning. Across 23 tasks against five strong baselines, the variants rank first on 18. The largest margins appear on Classical Arabic, and the authors credit these mainly to diacritic modeling. The result shows that modernizing the encoder backbone and treating script-specific detail carefully pays off for retrieval-oriented Arabic NLP.

Key points

  • Overall results: NeoAraBERT Mix averages 83.79% across 23 tasks, against 81.04% for the closest competitor. The variants win 18 tasks, place second on two, third on two and fourth on one.
  • Muradif (synonyms, no fine-tuning): Mix scores 87.03%. The best baseline, AraModernBERT, scores 77.33%, and AraBERTv2 scores 64.56%.
  • Classical Arabic:
    • APCD Era F1 is 53.71%, versus 46.21% for AraModernBERT and about 26–28% for AraBERTv2, ARBERTv2 and MARBERTv2.
    • Meter prediction reaches 85.34%.
    • WikiNews-2014 diacritization reaches 94.84%.
  • Tokenizer: a 60k undiacritized WordPiece vocabulary is extended with the top 5k diacritic-sequence tokens, for a 65k total.
  • Stemming: uses the D3 segmentation scheme with a [+] boundary marker and UD POS tags per morpheme.
  • Supporting tools: a Rust re-implementation of the CAMeL disambiguator (fast-disambig) and a hand-segmented fallback list of 5,000 frequent words.
  • Corpus cleaning: near-duplicate removal with ONIONS at roughly 80% similarity, keeping the higher-quality copy.
  • Dialect trade-off: continued training on dialect data alone cost MSA performance (the DA variant won no MSA tasks), whereas mixing dialect and MSA data gave a balanced model.

Interesting ideas

  • I1. Diacritic vocabulary from redacted text: to learn a vocabulary of diacritic patterns, they replaced every letter with one placeholder character and every digit with "1", so the tokenizer could only pick up the marks themselves; this cleanly separates a secondary signal from lexical content.
  • I2. Bare form plus annotation string: each word is emitted in its plain form followed by its diacritic sequence, with a placeholder for missing marks, so lookups stay canonical and the extra information is still present.
  • I3. Hand-made exception list for the analyzer: where the CAMeL analyzer misses subtle affixes, a hand-segmented list of 5,000 frequent words acts as a fallback, a cheap fix that targets the highest-frequency errors.
  • I4. POS-coherent group masking: masking units are whole words, plus adjacent segments merged when they share a POS, with nouns and verbs prioritized; this makes the model reconstruct meaningful units instead of fragments.
  • I5. Fine-tuning-free embedding probe: Muradif judges embeddings directly by whether synonyms land close together, isolating representation quality from fine-tuning effects.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): A similar redaction could let the en tokenizer learn a small inventory of casing, accent or punctuation-attachment patterns as separate tokens, keeping the lemma's canonical spelling intact.
  • I2 (tokens): Foreign or accented words in English text could be stored as a normalized form plus an attached feature string, so lemma lookup never fails on orthographic variants.
  • I3 (tokens): The morphology and lemma module could keep a curated override table for frequent words it misanalyses, ranked by corpus frequency so that a small list covers most errors.
  • I4 (learn): When training gparse or any masked pretraining step, tagger output could drive the masks over full content words and phrases, focusing learning signal where semantic roles live.
  • I5 (learn): A WordNet-synonym probe, paired with random-pair negative controls, could check whether Mova's lexical or frame representations are meaningful before any downstream training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓American University of BeirutBeirut · Lebanon34 works · 2,561 citations🔬Arab Center for Research and Policy StudiesDoha · Qatar8 works · 69 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksTokenization & segmentation · 119 works

Cite

Cite as Chadi Abou Chakra, Hadi Khaled Hamoud, Osama Rakan Al Mraikhat, Qusai Abu Obaida, Mohamad Ballout and Fadi Zaraket. 2026. NeoAraBERT: A Modern Foundation Model for Arabic Embeddings with Diacritics-Aware Tokenization and POS-Targeted Masking. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1293

@inproceedings{chakra2026neoarabert,
  title = {{NeoAraBERT: A Modern Foundation Model for Arabic Embeddings with Diacritics-Aware Tokenization and POS-Targeted Masking}},
  author = {Chadi Abou Chakra, Hadi Khaled Hamoud, Osama Rakan {Al Mraikhat}, Qusai Abu Obaida, Mohamad Ballout and Fadi Zaraket},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1293},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1293}
}

References

78 references, 47 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mourad Abbas and Kamel Smaili. 2005. Comparison of topic identification methods for arabic language. In Proceedings of International Conference on Recent Advances in Natural Language Processing, RANLP, pages 14–17. find
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and 1 others. 2021. Arbert & marbert: Deep bidirectional transformers for arabic. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 7088–7105.
  3. Qusai Abu Obaida, Digital Research Unit Arab Center for Research, and Policy Studies (U4RASD). 2025. U4rasd/curriculum_books_sft. Hugging Face Datasets. Version: main (commit 899b1f4). Accessed: 2026-01-05. find
  4. Murtadha Ahmed, Saghir Alfasly, Bo Wen, Jamal Ad-deen, Mohammed Ahmed, and Yunfeng Liu. 2024. Alclam: Arabic dialect language model. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 153–159.
  5. Karim Al-Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022. Curras + baladi: Towards a Levantine corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 769–778, Marseille, France. European Language Resources Association. ↗
  6. Zeina Aldallal, Sara Chrouf, Khalil Hennara, Mohamed Motaism Hamed, Muhammad Hreden, and Safwan AlModhayan. 2025. Sadeed: Advancing arabic diacritization through small language model. Preprint, arXiv:2504.21635. ↗
  7. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label arabic dataset for target-specific stance detection. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184.
  8. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. Araelectra: Pre-training text discriminators for arabic language understanding. In Proceedings of the sixth arabic natural language processing workshop, pages 191–195. ↗
  9. Jordan Ash and Ryan P Adams. 2020. On warm-starting neural network training. Advances in neural information processing systems, 33:3884–3894. ↗
  10. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609. ↗
  11. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. CoRR, abs/2001.08435. ↗
  12. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. ↗
  13. Yassine Benajiba and Paolo Rosso. 2007. Anersys 2.0: Conquering the ner task for the arabic language by combining the maximum entropy with pos-tag information. In IICAI, pages 1814–1823. find
  14. Gagan Bhatia, Abdellah El Mekki, Fakhraddin Alwajih, Muhammad Abdul-Mageed, and 1 others. 2025. Swan and arabicmteb: Dialect-aware, arabic-centric, cross-lingual, and cross-cultural embedding models and benchmarks. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4654–4670.
  15. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, and 1 others. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. ↗
  16. Lola Le Breton, Quentin Fournier, John Xavier Morris, Mariam El Mezouar, and Sarath Chandar. 2025. NeoBERT: A next generation BERT. Transactions on Machine Learning Research. Reproducibility Certification. ↗
  17. Amina Chouigui, Oussama Ben Khiroun, and Bilel Elayeb. 2021. An arabic multi-source news corpus: experimenting on single-document extractive summarization. Arabian Journal for Science and Engineering, 46(4):3925–3938. ↗
  18. Amina Chouigui, Oussama Ben Khiroun, and Bilel Elayeb. 2017. Ant corpus: An arabic news text collection for textual classification. In 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA), pages 135–142. IEEE. find
  19. Kamel Gaanoun, Abdou Mohamed Naira, Anass Allak, and Imade Benelallam. 2025. Darijabert: a step forward in nlp for the written moroccan dialect. International Journal of Data Science and Analytics, 20(2):917–929. ↗
  20. Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2020. Rethinking embedding coupling in pre-trained language models. arXiv preprint arXiv:2010.12821. ↗
  21. Abbas Ghaddar, Yimeng Wu, Sunyam Bagga, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Xinyu Duan, Zhefeng Wang, and 1 others. 2022. Revisiting pre-trained language models and their evaluation for arabic natural language processing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3135–3151.
  22. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. ↗
  23. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2475–2485. ↗
  24. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186. ↗
  25. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  26. Omar Einea, Ashraf Elnagar, and Ridhwan Al Debsi. 2019. Sanad: Single-label arabic news articles dataset for automatic text categorization. Data in Brief, 25:104076. ↗
  27. Mahmoud El-Haj, Paul Rayson, and Mariam Aboelezz. 2018. Arabic dialect identification in the context of bivalency and code-switching. In Proceedings of the 11th International Conference on Language Resources and Evaluation, Miyazaki, Japan., pages 3622–3627. European Language Resources Association.
  28. Ibrahim Abu El-Khair. 2016. 1.5 billion words arabic corpus. CoRR, abs/1611.04033. ↗
  29. Mohammed El-Razzaz, Mohamed Waleed Fakhr, and Fahima A Maghraby. 2021. Arabic gloss wsd using bert. Applied Sciences, 11(6):2567. ↗
  30. Ibrahim Abu Farha and Walid Magdy. 2020. From arabic sentiment analysis to sarcasm detection: The arsarcasm dataset. In The 4th Workshop on Open-Source Arabic Corpora and Processing Tools, pages 32–39. European Language Resources Association (ELRA).
  31. Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. 2021. Larger-scale transformers for multilingual masked language modeling. arXiv preprint arXiv:2105.00572. ↗
  32. Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, and 1 others. 2024. Olmo: Accelerating the science of language models. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pages 15789–15809. find
  33. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654. ↗
  34. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104.
  35. Mustafa Jarrar. 2021. The arabic ontology – an arabic wordnet with ontologically clean content. Applied Ontology, 16(1):1–26. ↗
  36. Mustafa Jarrar and Hamzeh Amayreh. 2019. An arabic-multilingual database with a lexicographic search engine. In Natural Language Processing and Information Systems, pages 234–246, Cham. Springer International Publishing. find
  37. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested Arabic named entity corpus and recognition using BERT. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3626–3636, Marseille, France. European Language Resources Association.
  38. Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammed Khalilia. 2023a. Salma: Arabic sense-annotated corpus and wsd benchmarks. arXiv preprint arXiv:2310.19029. ↗
  39. Mustafa Jarrar, Fadi A Zaraket, Tymaa Hammouda, Daanish Masood Alavi, and Martin Wählisch. 2023b. Lisan: Yemeni, iraqi, libyan, and sudanese arabic dialect corpora with morphological annotations. find
  40. In 2023 20th ACS/IEEE International Conference on Computer Systems and Applications (AICCSA), pages 1–7. find
  41. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. ↗
  42. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the association for computational linguistics, 8:64–77. ↗
  43. Mram Kahla, Zijian Győző Yang, and Attila Novák. 2021. Cross-lingual fine-tuning for abstractive Arabic text summarization. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 655–663, Held Online. INCOMA Ltd.
  44. Ruba Kharsa, Ashraf Elnagar, and Sane Yagi. 2024. Bert-based arabic diacritization: A state-of-the-art approach for improving text accuracy and pronunciation. Expert Systems with Applications, 248:123416. ↗
  45. Salima Lamsiyah, Kamyar Zeinalipour, Matthias Brust, Marco Maggini, Pascal Bouvry, Christoph Schommer, and 1 others. 2025. Arabicsense: A benchmark for evaluating commonsense reasoning in arabic with large language models. In Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4), pages 1–11.
  46. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  47. Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023. Xlm-v: Overcoming the vocabulary bottleneck in multilingual masked language models. arXiv preprint arXiv:2301.10472. ↗
  48. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  49. Abubakr Mohamed and Hamdy Mubarak. 2025. Advancing arabic diacritization: Improved datasets, benchmarking, and state-of-the-art models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 16718– 16730. ↗
  50. Otman Moussaoui and Yacine El Younnoussi. 2023. Pre-training two bert-like models for moroccan dialect: Morroberta and morrbert. In MENDEL, volume 29, pages 55–61. ↗
  51. NAMAA. 2025. Aramodernbert: Advanced arabic language model through trans-tokenization and modernbert architecture. https://huggingface.co/ NAMAA-Space/AraModernBert-Base-V1.0. Accessed: 2025-03-02. ↗
  52. Amal Nayouf, Tymaa Hammouda, Mustafa Jarrar, Fadi Zaraket, and Mohamad-Bassam Kurdy. 2023. Nâbra: Syrian Arabic dialects with morphological annotations. In Proceedings of ArabicNLP 2023, pages 12–23, Singapore (Hybrid). Association for Computational Linguistics. ↗
  53. Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Jan Hajic, Christopher D Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the twelfth language resources and evaluation conference, pages 4034–4043. ↗
  54. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  55. Pedro Javier Ortiz Su’arez, Laurent Romary, and Benoit Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703– 1714, Online. Association for Computational Linguistics.
  56. Pedro Javier Ortiz Su’arez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannheim. Leibniz-Institut f"ur Deutsche Sprache. ↗
  57. Jan Pomikálek. 2011. Removing boilerplate and duplicate content from web corpora. Ph.D. thesis, Masaryk university, Faculty of informatics, Brno, Czech Republic. find
  58. Faisal Qarah. 2024a. Egybert: A large language model pretrained on egyptian dialect corpora. arXiv preprint arXiv:2408.03524. ↗
  59. Faisal Qarah. 2024b. Saudibert: A large language model pretrained on saudi dialect corpora. arXiv preprint arXiv:2405.06239. ↗
  60. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316–1331. ↗
  61. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media. arXiv preprint arXiv:2007.13184. ↗
  62. Haitham Seelawi, Ahmad Mustafa, Hesham Al-Bataineh, Wael Farhan, and Hussein T Al-Natsheh. 2019. Nsurl-2019 shared task 8: Semantic question similarity in arabic. arXiv preprint arXiv:1909.09691. ↗
  63. Sakib Shahriar, Noora Al Roken, and Imran Zualkernan. 2023. Classification of arabic poetry emotions using deep learning. Computers, 12(5):89. ↗
  64. Noam Shazeer. 2020a. Glu variants improve transformer. arXiv preprint arXiv:2002.05202. ↗
  65. Noam Shazeer. 2020b. Glu variants improve transformer. ArXiv, abs/2002.05202. ↗
  66. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641–649.
  67. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. ↗
  68. Dongjie Yang, Zhuosheng Zhang, and Hai Zhao. 2023. Learning better masking for better language model pre-training. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7255–7267. find
  69. Waleed A Yousef, Omar M Ibrahime, Taha M Madbouly, and Moustafa A Mahmoud. 2019. Learning meters of arabic and english poems with recurrent neural networks: a step forward for language understanding and synthesis. arXiv preprint arXiv:1905.05700. ↗
  70. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063. ↗
  71. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and 1 others. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283–17297.
  72. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. Roformer: Enhanced transformer with rotary position embedding. Preprint, arXiv:2104.09864. ↗
  73. Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. Advances in neural information processing systems, 32. ↗
  74. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. ↗
  75. Micha l Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534, Portorož, Slovenia. European Language Resources Association (ELRA).
  76. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. ↗
  77. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, and 1 others. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2526–2547.
  78. Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2023. Should you mask 15% in masked language modeling? In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2985– 3000, Dubrovnik, Croatia. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chakra-2026-neoarabert-modern-foundation-model-arabic/