Elamur Research
☰
home / papers / 2026 / HalleluBERT: Let Every Token That Has Meaning Bear Its Weight

HalleluBERT: Let Every Token That Has Meaning Bear Its Weight

What it is about

The paper presents HalleluBERT, a RoBERTa-style encoder for Modern Hebrew in base (126M parameters) and large (357M) sizes, pre-trained from scratch on about 49.1 GB of deduplicated Hebrew web text (HeDC4) and Hebrew Wikipedia, with a 52k byte-level BPE vocabulary. Pre-training used 100k steps with an 8k global batch on a TPU pod (about 30 hours for base, about 6 days for large). It is evaluated on named entity recognition (BMC and token-level NEMO, micro-F1) and on sentiment classification (SMCD, macro-F1). Base reaches 93.33 on BMC and 87.06 on NEMO. Large has the best overall average (88.95 against 87.49 for XLM-RoBERTa large). The results come from a single seed, without confidence intervals, and the paper lists its own limits: no error analysis, no whole-word masking, and no control for register (Modern against Biblical Hebrew).

Ideas

For Mova

I1 is a measure for languages/he/12-mova.md, not built into anything. I2 goes into languages/he/02-script-and-spelling.md and the segmenter notes of tools/hbo. I3 goes into languages/he/10-historical-layers.md and tools/langseg.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC 4.0 (ELRA, LREC proceedings)" does not allow redistributing the text itself. Original — aclanthology.org/2026.lrec-1.236.pdf

Summary

Hebrew has several BERT-style encoders, but none is a RoBERTa model trained at scale or available in a large size. HalleluBERT addresses this with two RoBERTa encoders, base (126M parameters) and large (357M), pre-trained from scratch with fairseq on a TPUv4 pod. The training corpus combines the deduplicated HeDC4 web collection with a Hebrew Wikipedia dump, about 49.1 GB in total. Text is segmented with a 52k byte-level BPE vocabulary learned specifically on Hebrew data. Both models follow the standard RoBERTa schedule of 100k updates at a batch size of 8k. The author compares them with Hebrew-specific models (HeBERT, AlephBERT, AlephBERT-Gimmel, HeRo) and multilingual ones (XLM-RoBERTa, mmBERT). The benchmarks are two NER datasets, BMC and NEMO, and the SMCD sentiment corpus, and each model was tuned with a small grid search. HalleluBERT leads on NER and has the best mean score across all three tasks, with the large variant ranking first. Weights and tokenizer are released under the MIT license, giving Hebrew NLP an open, scaled-up baseline.

Key points

  • Pre-training data: HeDC4 (~47.5 GB, derived from mC4 and OSCAR22) plus Hebrew Wikipedia (~1.6 GB), shuffled before training.
  • Tokenizer: byte-level BPE with a 52k vocabulary, trained on a 20 GB Hebrew sample.
  • Training cost: about 30.2 hours for the base model and roughly 6 days for the large model, both in full precision. The 100k steps correspond to about 61 passes over the data.
  • NER results (micro-F1):
    • HalleluBERT base scores 93.33 on BMC and 87.06 on NEMO, the top NER average among base models (90.20).
    • HalleluBERT large reaches 88.70 on NEMO and a 90.96 NER average.
  • Sentiment and overall results:
    • On SMCD (macro-F1), AlephBERT leads the base models (83.66), while HalleluBERT large scores 84.91, the best among large models.
    • The overall three-task average is 88.95 for the large model and 87.83 for the base model, ranking first and second.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/3qdqexx4e9i2

@inproceedings{schmitt2026hallelubert,
  title = {{HalleluBERT: Let Every Token That Has Meaning Bear Its Weight}},
  author = {Schmitt, Raphael},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3qdqexx4e9i2},
  url = {https://doi.org/10.63317/3qdqexx4e9i2}
}

References

26 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. arXiv e-prints, page arXiv:2201.06642. ↗ our card
  2. Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from Modern Hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242– 2252, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
  3. Dan Bareket and Reut Tsarfaty. 2021. Neural Modeling for Named Entities and Morphology (NEMO2). Transactions of the Association for Computational Linguistics, 9:909–928. find our card
  4. Naama Ben-Mordecai. 2005. Hebrew named entity recognition. Master’s thesis, Department of Computer Science, Ben-Gurion University. find
  5. Branden Chan. 2020. XLM-RoBERTa: The multilingual alternative for non-english NLP. Library Catalog: towardsdatascience.com. find
  6. Branden Chan, Stefan Schweter, and Timo Möller. 2020. German’s next language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788–6796, Barcelona, Spain (Online). International Committee on Computational Linguistics. find our card
  7. Avihay Chriqui and Inbal Yahav. 2022. Hebert and hebemo: A hebrew bert model and a tool for polarity analysis and emotion recognition. IN-FORMS Journal on Data Science, 1(1):81–95. find
  8. Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. RobBERT: a Dutch RoBERTa-based Language Model. arXiv:2001.06286 [cs]. ArXiv: 2001.06286. ↗ our card
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  10. Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2022. Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all. find our card
  11. Henry He, Johann Frei, and Raphael Scheible-Schmitt. 2025. The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP. ISSN: 2693-5015. find
  12. Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David Patterson. 2023. TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. find
  13. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177–180, Prague, Czech Republic. Association for Computational Linguistics. find
  14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs]. ArXiv: 1907.11692. ↗
  15. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmbert: A modern multilingual encoder with annealed language learning. find
  16. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a Tasty French Language Model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, Online. Association for Computational Linguistics. find our card
  17. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. arXiv:1904.01038 [cs]. ArXiv: 1904.01038. ↗ our card
  18. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
  19. Raphael Scheible, Johann Frei, Fabian Thomczyk, Henry He, Patric Tippmann, Jochen Knaus, Victor Jaravine, Frank Kramer, and Martin Boeker. 2024. GottBERT: a pure German language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21237–21250, Miami, Florida, USA. Association for Computational Linguistics. find
  20. Raphael Scheible-Schmitt and Johann Frei. 2025. GeistBERT: Breathing life into German NLP. In Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Large Language Models, pages 42– 50, Varna, Bulgaria. INCOMA Ltd., Shoumen, BULGARIA. find
  21. Raphael Scheible-Schmitt, Henry He, and Armando B. Mendes. 2025. PortBERT: Navigating the depths of Portuguese language models. In Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Large Language Models, pages 59– 71, Varna, Bulgaria. INCOMA Ltd., Shoumen, BULGARIA. find
  22. Raphael Scheible-Schmitt and Stefan Schweter. 2025. Sindbert, the sailor: Charting the seas of turkish nlp. find
  23. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert:a hebrew large pre-trained language model to start-off your hebrew nlp application with. find our card
  24. Vitaly Shalumov and Harel Haskey. 2023. Hero: Roberta and longformer hebrew language models. find our card
  25. Cagri Toraman, Eyup Halit Yilmaz, Furkan Şahinu¸c, and Oguzhan Ozcelik. 2023. Impact of tokenization on language models: An analysis for turkish. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(4):1–21. find
  26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find our card