Elamur Research⌕
☰
home / papers / 2026 / SindBERT, the Sailor: Charting the Seas of Turkish NLP

SindBERT, the Sailor: Charting the Seas of Turkish NLP

Paper · 2026 · Raphael Schmitt and Stefan Schweter

Summary

Turkish is a morphologically rich, agglutinative language. It already has several transformer encoders, but no RoBERTa model had been pre-trained at full scale. Earlier RoBERTa attempts used small batches and short schedules. The authors fill this gap with SindBERT, trained from scratch in base and large sizes on about 312 GB of lightly filtered Turkish web text from mC4, OSCAR23 and Wikipedia. They follow the recipe of the German GottBERT, building a Turkish-specific byte-level BPE vocabulary and running the standard RoBERTa schedule in fairseq on TPUv4 hardware. After fine-tuning with Flair, the models were compared with Turkish and multilingual encoders on four tasks: PoS tagging, NER, offensive language detection and the TurBLiMP acceptability benchmark. SindBERT is competitive throughout, and the large variant ranks first on two of the four tasks. However, the larger size brings no consistent gain, a flat pattern also seen for XLM-R and EuroBERT. The authors read this as a sign that Turkish benchmarks are saturated. Pointing to the smaller but curated BERTurk, they argue that data quality and diversity matter more than raw volume. Both models are openly released under the MIT license.

Key points

  • Pre-training corpus: about 312 GB in total, made up of mC4 (~242 GB), OSCAR23 (~69 GB) and Wikipedia (~0.6 GB). The only filtering removed documents with broken character encodings, and no deduplication was done.
  • Tokenizer: a 52k-entry BPE vocabulary learned from a random 40 GB sample of the corpus.
  • Training setup: fairseq on a 128-core TPUv4 pod, in full fp32 precision, with 100k updates and a global batch of 8k.
    • SindBERT base took roughly 29.2 hours.
    • SindBERT large took about 6 days.
  • Evaluation tasks:
    • PoS tagging on five concatenated UD treebanks (Atis, BOUN, FrameNet, IMST, Tourism).
    • NER on WikiANN, using the Rahimi et al. splits.
    • Offensive language detection on OffensEval-TR 2020.
    • Linguistic acceptability on TurBLiMP.
  • Availability: the models are published in fairseq and Huggingface formats (huggingface.co/SindBERT). Stated limitations include web-only data, no control for dialect or register, and no systematic error analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technical University of MunichMunich · Germany55 works · 48,474 citations🎓University of FreiburgFreiburg im Breisgau · Germany6 works · 1,068 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Raphael Schmitt and Stefan Schweter. 2026. SindBERT, the Sailor: Charting the Seas of Turkish NLP. Proceedings of the Second Workshop Natural Language Processing for Turkic Languages (SIGTURK 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.sigturk-1.1

@inproceedings{schmitt2026sindbert,
  title = {{SindBERT, the Sailor: Charting the Seas of Turkish NLP}},
  author = {Raphael Schmitt and Stefan Schweter},
  booktitle = {Proceedings of the Second Workshop Natural Language Processing for Turkic Languages (SIGTURK 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.sigturk-1.1},
  url = {https://doi.org/10.18653/v1/2026.sigturk-1.1}
}

References

31 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages…
  2. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. Preprint, arXiv:2002.06305. find
  3. Henry He, Johann Frei, and Raphael Scheible-Schmitt. 2025. The word and the way: Strategies for domain-specific BERT pre-training in german medical NLP. ISSN: 2693-5015.
  4. Ezgi Başar, Francesca Padovani, Jaap Jumelet, and Arianna Bisazza. 2025. Turblimp: A turkish benchmark of linguistic minimal pairs. Preprint, arXiv:2506.13487.
  5. Tim Jansen, Yangling Tong, Victoria Zevallos, and Pedro Ortiz Suarez. 2022. Perplexed by quality: A perplexity-based method for adult and harmful content detection in multilingual heterogeneous web data. Preprint, arXiv:2212.10440. find
  6. Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, André Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, João Alves, Kevin El-Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei… — EuroBERT: Scaling Multilingual Encoders for European Languages
  7. Zihang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. 2021. Convbert: Improving bert with span-based dynamic convolution. Preprint, arXiv:2008.02496. find
  8. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tinybert: Distilling bert for natural language understanding. Preprint, arXiv:1909.10351.
  9. Çağrı Çöltekin. 2020. A corpus of turkish offensive language on social media. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6174–6184, Marseille, France.
  10. Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David Patterson. 2023. TPU v4: An optically reconfigurable supercomputer for machine learning with hardware suppo…
  11. Branden Chan. 2020. XLM-RoBERTa: The multilingual alternative for non-english NLP. Library Catalog: towardsdatascience.com. find
  12. Branden Chan, Stefan Schweter, and Timo Möller. 2020. German’s next language model. In Proceedings of the 28th International Conference on Computational Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Ric…
  13. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. A…
  14. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs]. ArXiv: 1907.11692.
  16. Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151–164, Florence, Italy. Association for Computational Linguistics.
  17. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmbert: A modern multilingual encoder with annealed language learning. Preprint, arXiv:2509.06888.
  18. Raphael Scheible, Johann Frei, Fabian Thomczyk, Henry He, Patric Tippmann, Jochen Knaus, Victor Jaravine, Frank Kramer, and Martin Boeker. 2024. GottBERT: a pure German language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21237–21250, Miami… find
  19. Büşra Marşan, Neslihan Kara, Merve Özçelik, Bilge Nas Arıcan, Neslihan Cesur, Aslı Kuzgun, Ezgi Sanıyar, Oğuzhan Kuyrukçu, and Olcay Taner Yildiz. 2021. Building the Turkish FrameNet. In Proceedings of the 11th Global Wordnet Conference, pages 118–125, University of South Africa (UNISA). Global W… find
  20. Raphael Scheible-Schmitt and Johann Frei. 2025. GeistBERT: Breathing life into German NLP. In Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Large Language Models, pages 42–50, Varna, Bulgaria. INCOMA Ltd., Shoumen, BUL-GARIA. find
  21. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a Tasty French Language Model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, O…
  22. Raphael Scheible-Schmitt, Henry He, and Armando B. Mendes. 2025. PortBERT: Navigating the depths of Portuguese language models. In Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Large Language Models, pages 59–71, Varna, Bulgaria. IN-COMA Lt… find
  23. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Eval…
  24. Stefan Schweter. 2020. Berturk - bert models for turkish. https://doi.org/10.5281/zenodo. 3770924. Version 1.0.0, Zenodo. — [doi:10.5281/zenodo](https://doi.org/10.5281/zenodo) ↗
  25. Stefan Schweter. 2025. BERTurk v2. https://doi.org/10.5281/zenodo.14963493. Version 2.0.0, Zenodo. — [doi:10.5281/zenodo.14963493](https://doi.org/10.5281/zenodo.14963493) ↗
  26. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. arXiv:1904.01038 [cs]. ArXiv: 1904.01038.
  27. Umut Sulubacak, Memduh Gokirmak, Francis Tyers, Çağrı Çöltekin, Joakim Nivre, and Gülşen Eryiğit. 2016. Universal Dependencies for Turkish. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3444–3454, Osaka, Japan. find
  28. The COLING 2016 Organizing Committee. find
  29. Şaziye Özateş, Tarık Tıraş, Efe Genç, and Esma Bilgin Tasdemir. 2024. Dependency annotation of Ottoman Turkish with multilingual BERT. In Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII), pages 188–196, St. Julians, Malta. Association for Computational Linguistics. find
  30. Nuri Tas. 2024. Roberturk: Adjusting roberta for turkish. Preprint, arXiv:2401.03515. — [arXiv:2401.03515](https://arxiv.org/abs/2401.03515) find
  31. A Cagri Toraman, Eyup Halit Yilmaz, Furkan Şahinu¸c, and Oguzhan Ozcelik. 2023. Impact of tokenization on language models: An analysis for turkish. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(4):1–21. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schmitt-2026-sindbert-sailor-charting-seas-turkish/