DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrew

What it is about

The paper presents DictaBERT, a BERT-architecture model for Modern Hebrew, pre-trained with masked language modelling only (next-sentence prediction removed) on about 3.8B tokens: 70% cleaned HeDC4 web text and 30% news, blogs, subtitles and novels. Three fine-tuned heads are released with it: DictaBERT-seg (prefix segmentation, 52K sentences), DictaBERT-morph (five classifiers for POS, proclitic functions, features, suffix presence and suffix features) and DictaBERT-heq (question answering on HeQ). Against mBERT, AlephBERT, AlephBERT-Gimmel, HeRo and mT5-Base it scores best on most tasks: segmentation 98.16 (AlephBERT-Gimmel 98.09), NER 87.01 (HeRo 86.79), sentiment 89.79 (HeRo 89.56), and QA F1 72.9, close to mT5-XL at 73.5 with about ten times the parameters. The paper is a model release; its evaluation tables are the reference numbers.

Ideas

I1. Constrain prefix segmentation by a table of legal proclitic sets keyed on the word's initial letters. The model applies only the valid function sets for a word's first letters at inference. A deterministic pre table in the segmenter can enforce the same rule; measure how many illegal splits it removes on a held-out set.
I2. Keep the morphology as separate outputs: POS, proclitic functions (several may stack), core features, suffix presence and suffix features. This is a usable layout for mapping Modern Hebrew tags to UD features, one slot per decision.
I3. Reproduce the multi-task segmentation and POS setup of Seker et al. (2021) on the same data before any claim about a Mova segmenter, and report it against the paper's Table 1 figures.

For Mova

I1 goes into languages/he/06-prepositions-particles.md (the legal prefix sets) and into the planned Modern Hebrew segmenter in languages/he/12-mova.md, following the table pattern of tools/hbo. I2 goes into languages/he/11-ud-analysis.md. I3 is a test for languages/he/12-mova.md. The model itself is neural and is only a teacher or a reference; nothing of it is a dependency.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2308.16687

Summary

Hebrew NLP already had several BERT-style encoders, including mBERT, HeBERT, AlephBERT, AlephBERTGimmel and HeRo. Fine-tuned mT5 models, which are much larger, had recently achieved the best results on most Hebrew benchmarks. The authors introduce DictaBERT, a standard-size BERT trained on about 3B words of real (non-synthetic) Hebrew-dominant text. The corpus combines a further-cleaned HeDC4 with news, blogs, subtitles and fiction. Their main changes are to data handling and pre-training. They train with MLM only, drop NSP and never mask words split into several word-pieces. Training examples are built from whole sentences, a [BLANK] token is randomly inserted, and any example containing [UNK] is discarded. DictaBERT achieves the top scores among comparably sized models on morphology, NER and sentiment. Its largest improvement is on HeQ question answering, where it roughly matches mT5-XL, a model with over ten times more parameters. The authors also publish three task-specific fine-tuned models that can be called directly through HuggingFace, giving developers ready-made Hebrew segmentation, morphological tagging and QA.

Key points

Pre-training setup: The tokenizer is WordPiece with a 128,000-token vocabulary and a custom pre-tokenizer for Hebrew quotation marks and apostrophes. The corpus has 3B words (3.8B tokens), of which 70% is cleaned HeDC4 (roughly 15% of it removed as noise) and 30% comes from other sources. Training ran in 3 phases on 4 A100 40GB GPUs for 9.4 days and processed about 35B tokens.
Morphology and classification results (Seker et al. setup): DictaBERT scores Seg 98.16, POS 96.25 and Features 96.1. It reaches 87.01 token F1 on NER using the NEMO corpus and 89.79 on sentiment analysis (Amram et al., 2018). Each of these slightly exceeds the strongest baseline, AlephBERTGimmel or HeRo.
QA results (HeQ, about 30K samples from GeekTime and Wikipedia): DictaBERT obtains EM 63.6 and F1 72.9. For comparison, mT5-XL (3.7B parameters) scores 63.4 and 73.5, and ChatGPT scores only 9.38 and 32.19. The authors also release their HeQ-fine-tuned mT5-XL.
Released fine-tuned models:
- DictaBERT-seg handles prefix (proclitic) segmentation and was trained on 52K human-corrected sentences (1.2M words).
- DictaBERT-morph predicts POS, features, proclitics and suffixes. It was trained on the UD Treebank (about 5K sentences) plus about 35K sentences from the IAHLT UD corpus.
- DictaBERT-heq is a QA model trained on HeQ.
License: The models are released under CC BY-SA 4.0 together with sample code.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2308.16687

@misc{shmidman2023dictabert,
  title = {{DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrew}},
  author = {Shmidman, Shaltiel and Shmidman, Avi and Koppel, Moshe},
  year = {2023},
  eprint = {2308.16687},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.16687}
}

References

14 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from Modern Hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242–2252, Santa The base model DictaBERT is available at https://huggingface.co/dicta-il/dictabert The fine-tuned segmentation model DictaBERT-seg is available at https://huggingface.co/dicta-il/ dictabert-seg The fine-tuned morphology model DictaBERT-morph is available at https://huggingface.co/dicta-il/ dictabert-morph The fine-tuned QA model DictaBERT-heq is available at https://huggingface.co/dicta-il/ dictabert-heq. Fe, New Mexico, USA. Association for Computational Linguistics. ↗ our card
Dan Bareket and Reut Tsarfaty. 2021. Neural Modeling for Named Entities and Morphology (NEMO2). Transactions of the Association for Computational Linguistics, 9:909–928. find our card
Avihay Chriqui and Inbal Yahav. 2022. HeBERT and HebEMO: A hebrew BERT model and a tool for polarity analysis and emotion recognition. INFORMS Journal on Data Science, 1(1):81–95. ↗ our card
Amir DN Cohen, Hilla Merhav Fine, Yoav Goldberg, and Reut Tsarfaty. 2023. Heq: a large and diverse hebrew reading comprehension benchmark. find our card
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ↗
Matan Eyal, Hila Noga, Roee Aharoni, Idan Szpektor, and Reut Tsarfaty. 2022. Multilingual sequence-to-sequence models for hebrew nlp. find our card
Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2023. Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all. find our card
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
Shoval Sade, Amit Seker, and Reut Tsarfaty. 2018. The Hebrew Universal Dependency treebank: Past present and future. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 133–143, Brussels, Belgium. Association for Computational Linguistics. find our card
Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert:a hebrew large pre-trained language model to start-off your hebrew nlp application with. find our card
Vitaly Shalumov and Harel Haskey. 2023. Hero: Roberta and longformer hebrew language models. arXiv:2304.11077. ↗ our card
Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast wordpiece tokenization. find
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. ↗
Amir Zeldes, Nick Howell, Noam Ordan, and Yifat Ben Moshe. 2022. A second wave of ud hebrew treebanking and cross-domain parsing. find our card