Elamur Research⌕
☰
home / papers / 2026 / UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu

UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu

Summary

The authors built a diagnostic benchmark that tests how well large language models have acquired Urdu grammar. Urdu has few digital resources, and fine-grained tests for it have been lacking until now. UrBLiMP consists of 5,696 minimal pairs. The sentences in each pair differ in only one grammatical feature, and the model must assign higher probability to the correct variant. The pairs cover ten morphosyntactic phenomena, including ergativity, various types of agreement, dative objects, honorific forms and word order. Examples were taken from the Urdu Treebank and from unannotated corpora, where they were found with regular expressions. The examples were then modified with rules and checked manually. Inter-annotator agreement is 96.1%. The authors tested twenty-one multilingual models, as well as GPT-4o, which was given grammatical hints in the prompt. GPT-4o had the highest average accuracy, 97.4%. However, all models struggle with free word order and with long-distance subject–verb agreement. The work shows how to build such tests for other morphologically rich low-resource languages, including Ukrainian.

Key points

  • UrBLiMP contains 5,696 pairs across ten phenomena, each divided into several paradigms. The methodology builds on SLING. The data are openly available on GitHub. For comparison, the Urdu portion of MultiBLiMP has only about 1,000 examples and tests only subject agreement.
  • GPT-4o with grammatical hints reaches 97.4% and comes close to human level on regular phenomena such as aspect marker compatibility and ergativity. The weakest points of all models are word order variation and long-distance subject–verb agreement.
  • The best open models have similar average scores: LLaMA-3-70B — 94.73, Alif-1.0-8B-Instruct — 93.25, gemma-3-12b-pt — 92.68, gemma-3-27b-pt — 92.42. A Wilcoxon test with Holm correction found no statistically significant difference between them.
  • In the Gemma series, accuracy grows with model size, but beyond 12B the gain practically disappears. On several paradigms, gemma-3-27b-pt outperforms 70B models.
  • The base (pretrained) Gemma-3 models of size 1B, 4B and 12B substantially outperform their instruction-tuned versions. Thus, instruction tuning can degrade the syntactic competence of smaller models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.29 · arXiv 2508.01006

@inproceedings{adeeba2026urblimp,
  title = {{UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu}},
  author = {Adeeba, Farah and Dillon, Brian and Sajjad, Hassan and Bhatt, Rajesh},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.29},
  eprint = {2508.01006},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.29}
}

References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ezgi Başar, Francesca Padovani, Jaap Jumelet, and Arianna Bisazza. 2025. Turblimp: A turkish benchmark of linguistic minimal pairs. Preprint, arXiv:2506.13487. find our card
  2. Miriam Butt and Jafar Rizvi. 2008. Tense and aspect in urdu. In Jan Fagerberg, David C. Mowery, and Richard R. Nelson, editors, Layers of Aspect, chapter 3, pages 43–53. Patricia Cabredo Hofherr and Brenda Laca. find
  3. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. find our card
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. find
  5. Toqeer Ehsan and Sarmad Hussain. 2021. Development and evaluation of an urdu treebank (cle-utb) and a statistical parser. Language Resources and Evaluation, 55(2):pp. 287–328. find
  6. Dubrovnik, Croatia. Association for Computational Linguistics. find
  7. IBM Granite Team. 2024. Granite 3.0 language models. find
  8. Yixiao Song, Kalpesh Krishna, Rajesh Bhatt, and Mohit Iyyer. 2022. SLING: Sino linguistic evaluation of large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4606–4634, Abu Dhabi, United Arab Emirates. Association for Computational Li… find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… find
  10. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051, Online and Punta Cana, Dominic… find our card
  11. Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, and Arianna Bisazza. 2025. Multiblimp 1.0: A massively multilingual benchmark of linguistic minimal pairs. Preprint, arXiv:2504.02768. find
  12. Daria Kryvosheieva and Roger Levy. 2024. Controlled evaluation of syntactic knowledge in multilingual language models. Preprint, arXiv:2411.07474. find
  13. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. find
  14. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry,… find our card
  15. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, and 1 others. 2022. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786. find our card
  16. Russell Scheinberg, Ameeta Agrawal, Amber Shore, and So Young Lee. 2025. Explain-then-process: Using grammar prompting to enhance grammatical acceptability judgments. Preprint, arXiv:2506.02302. find
  17. Taiga Someya and Yohei Oseki. 2023. JBLiMP: Japanese benchmark of linguistic minimal pairs. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1581–1594, Munief Hassan Tahir, Sana Shams, Layba Fiaz, Farah Adeeba, and Sarmad Hussain. 2025. Benchmarking the performance of p… find
  18. Ekaterina Taktasheva, Maxim Bazhukov, Kirill Koncha, Alena Fenogenova, and Ekaterina Artemova. 2024. Rublimp: Russian benchmark of linguistic minimal pairs. find
  19. Large Traversaal. 2025. Alif 1.0 8b instruct. https://huggingface.co/large-traversaal/ Alif-1.0-8B-Instruct. Accessed: 2025-07-24. ↗
  20. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392. find
  21. Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021. CLiMP: A benchmark for Chinese language model evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2784–2790, Online. Association fo… find
  22. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… find
  23. Jeremy Zehr and Florian Schwarz. 2018. Penncontroller for internet based experiments (IBEX). find

↑ back to top

This page in Markdown →