Elamur Research⌕
☰
home / papers / 2026 / A New State-of-the-Art BERT Model for Judeo-Arabic

A New State-of-the-Art BERT Model for Judeo-Arabic

Paper · 2026 · Elisha Rosensweig, Yitzchak Lindenbaum, Hillel Gershuni et al.

Summary

Judeo-Arabic is the family of Arabic varieties that Jewish communities across the Middle East and North Africa wrote in Hebrew characters from about the 8th century onward. It has a large medieval textual heritage but very little NLP support. Digital corpora are small, spelling varies widely, and Arabic-script tools can only handle it after error-prone transliteration.

The authors build JABERT, the first BERT encoder pretrained specifically for this language. The training corpus combines native Judeo-Arabic material with a far larger body of medieval Arabic from OpenITI. The OpenITI texts were converted into Hebrew script, with randomized spelling choices that imitate scribal habits. The tokenizer has a 128K WordPiece vocabulary and includes special tokens that mark gaps in damaged manuscripts.

For evaluation, the team created two new resources: expert-transcribed Cairo Genizah lines for masked-word prediction, and the first annotated Judeo-Arabic homograph test set. On homograph disambiguation, JABERT beats Arabic and multilingual BERTs, transliteration-based pipelines, and the Hebrew-script HeArBERT. It also reads mixed Hebrew–Arabic text directly. The model and the test set are released openly.

Key points

  • Training sources:
    • Princeton Genizah Project (~1.6M words)
    • Friedberg Judeo-Arabic Project (FJMS, ~3.1M words)
    • Quotations from Blau's Dictionary of Mediaeval Judaeo-Arabic Texts (~42K words)
    • OpenITI texts transliterated into Hebrew script (~1.2B words; authors who died 390–905 AH)
  • Tokenizer: 128K WordPiece vocabulary with two lacuna tokens. [ONEGAP] marks a single damaged word, and [GAP] marks a longer or indeterminate gap.
  • Pretraining: four NVIDIA A40 GPUs, 86,010 iterations, 106.56B tokens, in three phases (128-token sequences, then 512-token sequences).
  • New evaluation data:
    • 3,862 Cairo Genizah lines from Cambridge Taylor-Schechter boxes.
    • A homograph set of 16 words, 37 meanings and 1,119 samples, labelled by two experts. Samples where the experts disagreed were dropped.
  • Release: JABERT is available on HuggingFace (MiDRASH-ERC/JABERT) under CC BY 4.0.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Bar-Ilan UniversityRamat Gan · Israel141 works · 23,225 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksHebrew & Aramaic · 496 works

Cite

Cite as Elisha Rosensweig, Yitzchak Lindenbaum, Hillel Gershuni, Vered Raziel-Kretzmer, Daniel Caine and Avi Shmidman. 2026. A New State-of-the-Art BERT Model for Judeo-Arabic. Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/52qpsk7c5bfc

@inproceedings{rosensweig2026new,
  title = {{A New State-of-the-Art BERT Model for Judeo-Arabic}},
  author = {Elisha Rosensweig, Yitzchak Lindenbaum, Hillel Gershuni, Vered Raziel-Kretzmer, Daniel Caine and Avi Shmidman},
  booktitle = {Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026},
  year = {2026},
  doi = {10.63317/52qpsk7c5bfc},
  url = {https://doi.org/10.63317/52qpsk7c5bfc}
}

References

16 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Bashar Alhafni, Go Inoue, Christian Khairallah, and Nizar Habash. 2023. Advancements in Arabic grammatical error detection and correction: An empirical investigation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6430–6448, Singapore. Association for Computational Linguistics.
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  4. Kfir Bar, Nachum Dershowitz, Lior Wolf, Yackov Lubarsky, and Yaacov Choueka. 2015. Processing judeo-arabic texts. In 2015 First International Conference on Arabic Computational Linguistics (ACLing), pages 138–144. find
  5. Joshua Blau. 2006. A Dictionary of Mediaeval Judaeo-Arabic Texts. The Academy of the Hebrew Language; The Israel Academy of Sciences and Humanities. find
  6. Juan Moreno Gonzalez, Bashar Alhafni, and Nizar Habash. 2025. A tale of two scripts: Transliteration and post-correction for judeo-arabic.
  7. Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2023. Large pre-trained models with extra-large vocabularies: A contrastive analysis of Hebrew BERT models and a new one to outperform them all. ↗
  8. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  9. Princeton Geniza Project. 2021. Transcription history and conventions. Google Docs. Archived at https://web.archive.org/web/2026 0204094447/https://docs.google.com/document/d/1pyFajrGRYTzRsb2yFjfX0X mNqpz1kUO9mhlISGwfpxk/edit?tab=t.0. Accessed March 25, 2025. ↗
  10. Aviad Rom and Kfir Bar. 2024. Training a bilingual language model by mapping tokens onto a shared character space. arXiv preprint arXiv:2402.16065. ↗
  11. Maxim Romanov. 2023. OpenITI mARkdown. Accessed April 7, 2025. find
  12. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast wordpiece tokenization. find
  13. Ori Terner, Kfir Bar, and Nachum Dershowitz. 2020. Transliteration of Judeo-Arabic texts into Arabic script using recurrent neural networks. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 85–96, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  14. Daniel Weisberg Mitelman, Nachum Dershowitz, and Kfir Bar. 2024. Code-switching and back-transliteration using a bilingual model. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1501–1511, St. Julian’s, Malta. Association for Computational Linguistics. 10.
  15. Language Resource References Romanov, Maxim G. and Miller, Matthew Thomas and Savant, Sarah Bowen and Verkinderen, Peter. 2024. OpenITI: a Machine-Readable Corpus of Islamicate Texts. Zenodo, 2023.1.8. Accessed September 25, 2024. find
  16. Rustow, Marina and Koeser, Rebecca Sutton. 2025. Princeton Geniza Project. Center for Digital Humanities at Princeton, 4.28.0. Accessed March 25, 2025. Text data retrieved via https://github.com/princetongenizalab/pgp-text. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rosensweig-2026-new-state-art-bert-model/