The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP (2026 · 2 citations)
German clinical NLP lacks strong pre-trained encoders. The existing German biomedical models either use older architectures or were trained on small amounts of data. The authors built ChristBERT, a family of RoBERTa-based models that extends the earlier GeistBERT. To compensate for the scarcity of German clinical…
SindBERT, the Sailor: Charting the Seas of Turkish NLP (2026 · 1 citation)
Turkish is a morphologically rich, agglutinative language. It already has several transformer encoders, but no RoBERTa model had been pre-trained at full scale. Earlier RoBERTa attempts used small batches and short schedules. The authors fill this gap with SindBERT, trained from scratch in base and large sizes on…
HalleluBERT: Let Every Token That Has Meaning Bear Its Weight (2026)
The paper presents HalleluBERT, a RoBERTa-style encoder for Modern Hebrew in base (126M parameters) and large (357M) sizes, pre-trained from scratch on about 49.1 GB of deduplicated Hebrew web text (HeDC4) and Hebrew Wikipedia, with a 52k byte-level BPE vocabulary.

Co-authors here

1 co-author with a page, the most papers together first.

Venues

Where their works here were published, the most works first, with the editions of those that have a page.

Papers of our library cite their works 3 times (of the 149,439 papers whose bibliographies we read); 2 citing papers with a page here →.

Updated 31 October 2026