Elamur Research⌕
☰
home / papers / 2026 / Serbian SuperGLUE: Towards an Evaluation Benchmark for South Slavic Language Models

Serbian SuperGLUE: Towards an Evaluation Benchmark for South Slavic Language Models

Paper · 2026 · Mitar Perovic and Teodora Mihajlov

Summary

Serbian, and BCMS languages more broadly, have no standard way to test language models on demanding understanding tasks. The only regional suite, BENCHić, covers simpler problems such as NER and sentiment. The authors fill this gap by adapting the English SuperGLUE into Serbian. They chose Gemini Flash 2.5 as the translator after annotators compared five LLMs, then applied automatic post-editing. Some tasks needed other routes: COPA was taken from BENCHić, ReCoRD was built from existing resources, and WiC was left out because no Serbian lexicon is available. Diagnostic subsets were checked by hand.

The authors fine-tuned seven encoder models on the resulting seven-task suite, comparing Serbian-specific and multilingual models. Models built only for Serbian did not consistently beat multilingual ones. Results on small datasets swung widely between random seeds, so the authors argue that multi-seed reporting is essential in low-resource evaluation. The benchmark, code and checkpoints are released openly as a shared base for future South Slavic NLU work.

Key points

  • The suite has seven tasks: BoolQ, CB, COPA, MultiRC, ReCoRD, RTE and WSC. They span QA, NLI and coreference, and WiC is excluded.
  • Seven encoders were evaluated, including BERTić, Jerteh, mmBERT, XLM-R-BERTić and TeslaXLM.
    • mmBERT scored best on RTE at 75.7%, and XLM-R-BERTić led on BoolQ at 82.0%.
    • BERTić was strong on MultiRC (F1a 65.8%) and on BoolQ (77.1%).
    • Jerteh (355M) did well on CB but failed on MultiRC.
  • Results varied sharply across seeds. XLM-R-BERTić on RTE ranged from 52.7% to 79.8% (σ = 12.7%). The authors recommend reporting results over at least 5 seeds and using a learning rate of 5×10⁻⁶ for datasets under 1,000 examples.
  • ReCoRD proved easiest, with all models at 94–96%, which likely reflects simpler generated questions.
  • WSC did not separate the models, which all scored about 63–65%. The authors cite three causes:
    • only 554 training examples;
    • roughly 65% positive labels;
    • translation artifacts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Mitar Perovic and Teodora Mihajlov. 2026. Serbian SuperGLUE: Towards an Evaluation Benchmark for South Slavic Language Models. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.30

@inproceedings{perovic2026serbian,
  title = {{Serbian SuperGLUE: Towards an Evaluation Benchmark for South Slavic Language Models}},
  author = {Mitar Perovic and Teodora Mihajlov},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.30},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.30}
}

References

52 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yvonne Adesam, Aleksandrs Berdicevskis, and Felix Morger. 2020. Swedishglue–towards a swedish test set for evaluating natural language understanding models. find
  2. Maksim Aparovich, Volha Harytskaya, Vladislav Poritski, Oksana Volchek, and Pavel Smrz. 2025. Belarusianglue: Towards a natural language understanding benchmark for belarusian. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5… find
  3. Marta Bañón, Miquel Esplà-Gomis, Mikel L Forcada, Cristian García-Romero, Taja Kuzman, Nikola Ljubešić, Rik Van Noord, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Peter Rupnik, and 1 others. 2022. Macocu: Massive collection and curation of monolingual and bilingual data: focus on under-resourced la… find
  4. Vuk Batanović, Nikola Ljubešić, Tanja Samardžić, and Tomaž Erjavec. 2023. Serbian linguistic training corpus SETimes.SR 2.0. Slovenian language resource repository CLARIN.SI. find
  5. Damir Ćavar and Dunja Brozović Rončević. 2012. Riznica: the croatian language corpus. Prace filologiczne, 63:51–65. find
  6. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.
  7. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. In Proceedings of Sinn und Bedeutung, volume 23, pages 107–124. Data can be found at https://github.com/mcdm/CommitmentBank/. ↗
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
  9. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International conference on machine learning, pages 4411–4421. PMLR. find
  10. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:…
  11. Taja Kuzman, Peter Rupnik, and Nikola Ljubešić. 2023. Get to know your parallel data: Performing English variety and genre classification over MaCoCu corpora. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 91–103, Dubrovnik, Croatia. Association for Com… find
  12. Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. KR, 2012(13th):3. find
  13. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, and 1 others. 2020. Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation. arXiv preprint arXiv:2004.01401.
  14. Noémi Ligeti-Nagy, Gergő Ferenczi, Enikő Héja, László János Laki, Noémi Vadász, Zijian Győző Yang, and Tamás Váradi. 2024. Hulu: Hungarian language understanding benchmark kit. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluati… find
  15. Nikola Ljubešić. 2021. Text collection for training the BERTić transformer model BERTić-data. Slovenian language resource repository CLARIN.SI. find
  16. Nikola Ljubešić and Tomaž Erjavec. 2021. Montenegrin web corpus meWaC 1.0. Slovenian language resource repository CLARIN.SI. find
  17. Nikola Ljubešić and Filip Klubička. 2014. {bs,hr,sr}WaC - web corpora of Bosnian, Croatian and Serbian. In Proceedings of the 9th Web as Corpus Workshop (WaC-9), pages 29–35, Gothenburg, Sweden. Association for Computational Linguistics. find
  18. Nikola Ljubešić and Taja Kuzman. 2024. Classla-web: comparable web corpora of south slavic languages enriched with linguistic and genre annotation. arXiv preprint arXiv:2403.12721. find
  19. Nikola Ljubešić and Davor Lauc. 2021. BERTić – the transformer language model for Bosnian, Croatian, Montenegrin and Serbian. In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, pages 37–42. Association for Computational Linguistics. find
  20. Nikola Ljubešić, Mirjana Starović, Taja Kuzman, and Tanja Samardžić. 2022. Choice of plausible alternatives dataset in serbian COPA-SR. Slovenian language resource repository CLARIN.SI. find
  21. Nikola Ljubešić, Vít Suchomel, Peter Rupnik, Taja Kuzman, and Rik van Noord. 2024a. Language models on a diet: Cost-efficient development of encoders for closely-related languages via additional pretraining. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced L… find
  22. Nikola Ljubešić, Vít Suchomel, Peter Rupnik, Taja Kuzman, and Rik van Noord. 2024b. Language models on a diet: Cost-efficient development of encoders for closely-related languages via additional pretraining. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced L… find
  23. Nikola Ljubešić, Filip Klubička, Željko Agić, and Ivo-Pavao Jazbec. 2016. New inflectional lexicons and training corpora for improved morphosyntactic annotation of croatian and serbian. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, F… find
  24. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmBERT: A modern multilingual encoder with annealed language learning. Preprint, arXiv:2509.06888.
  25. Ulfeta A Marovac, Aldina R Avdić, and Nikola Lj Milošević. 2023. A survey of resources and methods for natural language processing of serbian language. arXiv preprint arXiv:2304.05468. — [arXiv:2304.05468](https://arxiv.org/abs/2304.05468) find
  26. Michal Mochtak, Peter Rupnik, and Nikola Ljubešić. 2024. The ParlaSent multilingual training dataset for sentiment identification in parliamentary proceedings. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024… find
  27. Pedro Javier Ortiz Su’arez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannhei… find
  28. Tomás Freitas Osório, Bernardo Leite, Henrique Lopes Cardoso, Luís Gomes, João Rodrigues, Rodrigo Santos, and António Branco. 2024. Portulan extraglue datasets and models: Kick-starting a benchmark for the neural processing of portuguese. In Proceedings of the 17th Workshop on Building and Using Com… find
  29. Edvin Pakoci, Branislav Popović, and Darko Pekar. 2019. Using morphological data in language modeling for serbian large vocabulary speech recognition. Computational intelligence and neuroscience, 2019(1):5072918. find
  30. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
  31. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association…
  32. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, and 1 others. 2018. Improving language understanding by generative pre-training. find
  33. Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151–164, Florence, Italy. Association for Computational Linguistics.
  34. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90–95. find
  35. Peter Rupnik, Taja Kuzman, and Nikola Ljubešić. 2023. Benchić-lang: A benchmark for discriminating between bosnian, croatian, montenegrin and serbian. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 113–120. find
  36. Tanja Samardžić, Mirjana Starović, Željko Agić, and Nikola Ljubešić. 2017. Universal Dependencies for Serbian in comparison with Croatian and other Slavic languages. In Proceedings of the 6th Workshop on Balto-Slavic Natural Language Processing, pages 39– 44, Valencia, Spain. Association for Com… find
  37. Tatiana Shavrina, Alena Fenogenova, Emelyanov Anton, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Evlampiev. 2020. Russiansuperglue: A russian language understanding evaluation benchmark. In Proceedings of the 2020 conference o… find
  38. Mihailo Škorić. 2024a. Novi jezički modeli za srpski jezik. Infoteka, 24. ArXiv:2402.14379. — [arXiv:2402.14379](https://arxiv.org/abs/2402.14379) find
  39. Mihailo Škorić. 2024b. Novi jezički modeli za srpski jezik. Infoteka, 24. find
  40. Mihailo Škorić and Nikola Janković. 2024. New textual corpora for serbian language modeling. arXiv preprint arXiv:2405.09250. — [arXiv:2405.09250](https://arxiv.org/abs/2405.09250) find
  41. Ranka Stanković, Cvetana Krstev, Branislava Šandrih Todorović, Dusko Vitas, Mihailo Skoric, and Milica Ikonić Nešić. 2022. Distant reading in digital humanities: Case study on the Serbian part of the EL-TeC collection. In Proceedings of the Thirteenth Language Resources and Evaluation Conference… find
  42. Ranka Stankovic, Branislava Šandrih, Cvetana Krstev, Miloš Utvić, and Mihailo Skoric. 2020. Machine learning and deep neural network-based lemmatization and morphosyntactic tagging for Serbian. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3954–3962, Marseille, F… find
  43. European Language Resources Association. and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32.
  44. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
  45. Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang, and Zhenzhong Lan. 2023. Superclue: A comprehensive chinese large language model benchmark. arXiv preprint arXiv:2307.15020.
  46. Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, and 1 others. 2025. Mmlu-prox: A multilingual benchmark for advanced large language model evaluation. arXiv preprint arXiv:2503.10497.
  47. Aleš Žagar and Marko Robnik-Šikonja. 2022. Slovene superglue benchmark: translation and evaluation. arXiv preprint arXiv:2202.04994. find
  48. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885.
  49. Minh-Nam Tran, Phu-Vinh Nguyen, Long Nguyen, and Dien Dinh. 2024. ViGLUE: A Vietnamese general language understanding benchmark and analysis of Vietnamese language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4174–4189, Mexico City, Mexico. Association for… find
  50. Matej Ulčar and Marko Robnik-Šikonja. 2020. FinEst BERT and CroSloEngual BERT: Less is more in multilingual models. In Text, Speech, and Dialogue (TSD 2020), volume 12284 of Lecture Notes in Computer Science, pages 104–118. Springer. find
  51. Gorka Urbizu, Iñaki San Vicente, Xabier Saralegi, Rodrigo Agerri, and Aitor Soroa. 2022. Basqueglue: A natural language understanding benchmark for basque. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1603– 1612. find
  52. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/perovic-2026-serbian-superglue-evaluation-benchmark-south/