Elamur Research⌕
☰
home / papers / 2025 / TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs

TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs

Summary

The authors created TurBLiMP, the first set of linguistic minimal pairs for Turkish. It makes it possible to test the grammatical knowledge of monolingual and multilingual language models. The benchmark covers 16 grammatical phenomena, from anaphor agreement and argument structure to NPIs, ellipsis and island constraints. Each phenomenon has 1000 pairs in which one sentence is acceptable and the other is not. The pairs were created with a hybrid method adapted to the specifics of Turkish. In addition, the authors designed experimental paradigms that vary word order and the type of subordinate construction. In Turkish, subordination is expressed mainly through suffixes. For each pair, the authors compared the log-probabilities of the two sentences in 13 models. The results were compared against newly collected acceptability judgments from native speakers. Even the strongest LLMs make errors where humans answer confidently. Models respond to non-canonical word order and complex morphology differently from humans. The work extends BLiMP-style testing to an agglutinative language with free word order. It can therefore serve as a model for other morphologically rich languages.

Key points

  • Size: 16 phenomena with 1000 pairs each, 16 000 in total. Some of the phenomena are specific to Turkish, such as suspended affixation. Each phenomenon has one paradigm, except Argument Structure, which has two: transitive and ditransitive.
  • 13 models were tested. Larger models are generally more accurate, but there are exceptions. Much smaller monolingual models (in particular BERTurk) often outperform larger multilingual ones and are on par with the best of them.
  • Ellipsis (backward gapping) remains difficult for models of every size and architecture. This is where model behavior diverges most noticeably from human judgments.
  • The experimental paradigms test two parameters: word order (Turkish allows all six permutations of subject, object and verb) and different types of subordination (-DIK, finite, adverbial clauses). Even the best models lose accuracy on them, which the standard evaluation does not reveal.
  • Instruct variants of the models were not evaluated because, for example, Gemma 3 (4B) Instruct is on average 4.4% worse than the base version. The data are open: github.com/ezgibasar/turblimp.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

14 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 12 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.834

@inproceedings{basar2025turblimp,
  title = {{TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs}},
  author = {Başar, Ezgi and Padovani, Francesca and Jumelet, Jaap and Bisazza, Arianna},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.834},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.834}
}

References

54 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmet Afşın Akın and Mehmet Dündar Akın. 2007. Zemberek, an open source nlp framework for turkic languages. Structure, 10:1–5. find
  2. Bjarki Ármannsson, Finnur Ágúst Ingimundarson, and Einar Freyr Sigurðsson. 2025. An Icelandic linguistic benchmark for large language models. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT… find
  3. Catherine Arnett and Benjamin Bergen. 2025. Why do language models perform worse for morphologically complex languages? In Proceedings of the 31st International Conference on Computational Linguistics, pages 6607–6623, Abu Dhabi, UAE. Association for Computational Linguistics. find
  4. Z. Ceyda Arslan-Kechriotis. 2009. Referentiality in turkish: NP/DP. In Essays on Turkish Linguistics: Proceedings of the 14th International Conference on Turkish Linguistics, 6-8 August 2008, pages 83–92, Wiesbaden. Harrassowitz Verlag. find
  5. Duygu Ataman, Matteo Negri, Marco Turchi, and Marcello Federico. 2017. Linguistically motivated vocabulary reduction for neural machine translation from turkish to english. The Prague Bulletin of Mathematical Linguistics, 108:331 – 342. find
  6. Cem Bozşahin. 2000. Gapping and word order in Turkish. In Proceedings of the 10th International Conference on Turkish Linguistics, pages 58–66. find
  7. Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K. Bergen. 2024. Goldfish: Monolingual language models for 350 languages. Preprint. find
  8. Nadezhda Chirkova and Vassilina Nikoulina. 2024. Zero-shot cross-lingual transfer in instruction tuning of large language models. In Proceedings of the 17th International Natural Language Generation Conference, pages 695–708, Tokyo, Japan. Association for Computational Linguistics. find
  9. Noam Chomsky. 1965. Aspects of the Theory of Syntax, 50 edition. The MIT Press, Cambridge, MA, USA. find
  10. Noam Chomsky. 1981. Lectures on Government and Binding. Number 9 in Studies in Generative Grammar. Foris Publications, Dordrecht. find
  11. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. find
  12. The SIGMORPHON 2016 shared Task— Morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 10–22, Berlin, Germany. Association for Computational Linguistics. find
  13. Fred J. Damerau. 1964. A technique for computer detection and correction of spelling errors. Commun. ACM, 7(3):171–176. find
  14. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac,… find our card
  15. Mürvet Enç. 1991. The semantics of specificity. Linguistic Inquiry, 22(1):1–25. find
  16. Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. On the relation between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 316–327, Brussels,… find
  17. Aslı Göksel and Celia Kerslake. 2005. Turkish: A Comprehensive Grammar. Comprehensive grammars. Routledge. find
  18. Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024. Unpacking tokenization: Evaluating text compression and its correlation with model performance. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2274–2286, Bangkok, Thailand. Assoc… find
  19. Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy. 2024. Language models align with human judgments on key grammatical constructions. Proceedings of the National Academy of Sciences, 121(36):e2400917121. find
  20. Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Duygu Ataman, and Lonneke Van Der Plas. 2025. Evaluating morphological compositional generalization in large language models. In Proceedings of the 2025 Conference of the Nations of the Amer… find
  21. Jaap Jumelet and Dieuwke Hupkes. 2018. Do language models understand anything? on the ability of LSTMs to understand negative polarity items. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 222–231, Brussels, Belgium. Association for C… find
  22. Jaap Jumelet, Leonie Weissweiler, and Arianna Bisazza. 2025. Multiblimp 1.0: A massively multilingual benchmark of linguistic minimal pairs. Preprint, arXiv:2504.02768. find
  23. Carina Kauf and Anna Ivanova. 2023. A better way to do masked language model scoring. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 925–935, Toronto, Canada. Association for Computational Linguistics. find
  24. H. Toprak Kesgin, M. Kaan Yuce, Eren Dogan, M. Ege-men Uzun, Atahan Uz, H. Emre Seyrek, Ahmed Zeer, and M. Fatih Amasyali. 2024. Introducing cosmosgpt: Monolingual training for turkish language models. arXiv preprint arXiv:2404.17336. — [arXiv:2404.17336](https://arxiv.org/abs/2404.17336) find
  25. Jaklin Kornfilt. 2003a. Scrambling, Subscrambling, and Case in Turkish, chapter 6. John Wiley & Sons, Ltd. find
  26. Jaklin Kornfilt. 2003b. Subject Case in Turkish nominalized clauses, pages 129–216. De Gruyter Mouton, Berlin, Boston. find
  27. Daria Kryvosheieva and Roger Levy. 2025. Controlled evaluation of syntactic knowledge in multilingual language models. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 402–413, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  28. Wei Qi Leong, Jian Gang Ngui, Yosephine Susanto, Hamsawardhini Rengarajan, Kengatharaiyer Sarveswaran, and William Chandra Tjhi. 2023. Bhasa: A holistic southeast asian linguistic and cultural evaluation suite for large language models. Preprint, arXiv:2309.06085. find
  29. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. find our card
  30. Yikang Liu, Yeting Shen, Hongao Zhu, Lilong Xu, Zhiheng Qian, Siyuan Song, Kejia Zhang, Jialong Tang, Pei Zhang, Baosong Yang, Rui Wang, and Hai Hu. 2024. Zhoblimp: a systematic assessment of language models with linguistic minimal pairs in chinese. Preprint, arXiv:2411.06096. find
  31. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. Eurollm: Multilingual language m… find our card
  32. Meta. 2024. Llama 3.1: Open and efficient foundation language models. [Software]. Version 3.1. find
  33. Kanishka Misra. 2022. minicons: Enabling flexible behavioral and representational analyses of transformer language models. arXiv preprint arXiv:2203.13112. find
  34. Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, and Tal Linzen. 2020. Cross-linguistic syntactic evaluation of word prediction models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5523–5539, Online. Association for Comput… find
  35. Mine Nakipoğlu, Berna A. Uzundağ, and F. Nihan Ke-trez. 2023. Analogy is indispensable but rule is a must: Insights from turkish. Journal of Child Language, 50(2):437–463. find
  36. Hyunji Hayley Park, Katherine J. Zhang, Coleman Haley, Kenneth Steimel, Han Liu, and Lane Schwartz. 2021. Morphology matters: A multilingual language modeling analysis. Transactions of the Association for Computational Linguistics, 9:261–276. find
  37. Adam Pauls and Dan Klein. 2012. Large-scale syntactic language modeling with treelets. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 959–968, Jeju Island, Korea. Association for Computational Linguistics. find
  38. Stefan Schweter. 2020. Berturk - bert models for turkish. find
  39. Kutay Serova. 2019. Head movement, suspended affixation, and the turkish clausal spine. Proceedings of the Workshop on Turkic and Languages in Contact with Turkic, 4:89. find
  40. Dan I. Slobin and Thomas G. Bever. 1982. Children use canonical sentence schemas: A crosslinguistic study of word order and inflections. Cognition, 12(3):229– 265. find
  41. Taiga Someya and Yohei Oseki. 2023. JBLiMP: Japanese benchmark of linguistic minimal pairs. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1581–1594, Dubrovnik, Croatia. Association for Computational Linguistics. find
  42. Siyuan Song, Jennifer Hu, and Kyle Mahowald. 2025. Language models fail to introspect about their knowledge of language. Preprint, arXiv:2503.07513. find
  43. Yixiao Song, Kalpesh Krishna, Rajesh Bhatt, and Mohit Iyyer. 2022. SLING: Sino linguistic evaluation of large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4606–4634, Abu Dhabi, United Arab Emirates. Association for Computational Li… find
  44. Michelle Suijkerbuijk, Zoë Prins, Marianne de Heer Kloots, Willem Zuidema, and Stefan L. Frank. 2025. Blimp-nl: A corpus of dutch minimal pairs and acceptability judgments for language model evaluation. Computational Linguistics, pages 1–39. find
  45. Ekaterina Taktasheva, Maxim Bazhukov, Kirill Kon-cha, Alena Fenogenova, Ekaterina Artemova, and Vladislav Mikhailov. 2024. RuBLiMP: Russian benchmark of linguistic minimal pairs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9268–9299, Miami, Florid… find
  46. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  47. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179… find
  48. Utku Türk, Furkan Atmaca, Şaziye Betül Özateş, Gözde Berk, Seyyit Talha Bedir, Abdullatif Köksal, Balkız Öztürk Başaran, Tunga Güngör, and Arzucan Özgür. 2022. Resources for turkish dependency parsing: introducing the boun treebank and the boat annotation tool. Lang. Resour. Eval., 56(1):259–307. find
  49. Héctor Javier Vázquez Martínez, Annika Heuser, Charles Yang, and Jordan Kodner. 2023. Evaluating neural language models as cognitive models of language acquisition. In Proceedings of the 1st Gen-Bench Workshop on (Benchmarking) Generalisation in NLP, pages 48–64, Singapore. Association for Computati… find
  50. Sinan Çakır. 2016. Island constraints and adjunct & argument asymmetry in turkish. Dilbilim Araştır-maları Dergisi, 27:1–1. find
  51. Sumru A. Özsoy. 2009. Argument structure, animacy, syntax and semantics of passivization in Turkish: A corpus-based approach, pages 259–279. John Benjamins Publishing Company. find
  52. A Our experimental paradigms only target minimal pairs belonging to Argument Structure Transitive and Argument Structure Ditransitive phenomena. Below is an example of the transitive baseline. (3) find
  53. Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021. CLiMP: A benchmark for Chinese language model evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2784–2790, Online. Association fo… find
  54. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2025. Qwen2.5 technical report. arXiv preprint arXi… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/ar-2025-turblimp-turkish-benchmark-linguistic-minimal/