Elamur Research⌕
☰
home / papers / 2026 / Tracking the evolution of LLM capabilities for Belarusian with OpenAI Evals

Tracking the evolution of LLM capabilities for Belarusian with OpenAI Evals

Paper · 2026 · Vladislav Poritski, Oksana Volchek, Maksim Aparovich et al.

Summary

The paper asks how much LLM performance on Belarusian has improved since 2023, when eight Belarusian tasks were added to OpenAI's Evals framework. Those tasks were designed to be hard for models of that time. Because the original instances have been public as dev data, newer models might have seen them in training. The authors therefore built fresh, non-overlapping test sets using the original construction recipe.

They evaluated a range of systems against GPT-3.5 Turbo as a 2023 reference and against trivial one-line baselines. These included o4-mini, the GPT-4.1 and GPT-4o families, Claude 3.7 Sonnet, Gemini 2.5 Flash, and open-weight models (Gemma 3 27B, gpt-oss-120b, Qwen3-235B-A22B). Accuracy was the metric, and the authors followed it with a detailed error analysis.

Progress turned out to be large but uneven. Some tasks are close to saturated, while others barely exceed trivial baselines. Persistent weaknesses include:

  • accepting non-words as real vocabulary;
  • converting modern spelling into taraškievica.

The authors suggest these gaps reflect interference from Russian. The datasets and generated model outputs are publicly released.

Key points

  • The eight tasks are Syllable count, Numerals, Lexicon, Orthography, Rhyme, Inflectional analogies, Grammar, and round-trip Belarusian⇒Russian⇒Belarusian Translation. All are scored by accuracy.
  • Dev and test scores track each other almost linearly (y ≈ 1.01x − 0.03, r² ≈ 0.94), so there is no clear sign of contamination. Orthography and Rhyme test sets proved harder, partly due to items that defeat the rule-based Taraskevizatar tool.
  • In Lexicon, general-purpose models produce far more false positives than false negatives and gain little over the "always Y" baseline.
  • In Translation, GPT-4.1 returned the identical input sentence in only 4% of cases. Its edits showed Russian-influenced word choices and occasional grammatical errors.
  • Models were tested between May and October 2025 at temperatures 0.0 and 1.0, reporting the best run. Data and completions are on Zenodo (doi:10.5281/zenodo.18163825).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Brno University of TechnologyBrno · Czechia11 works · 32,326 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Vladislav Poritski, Oksana Volchek, Maksim Aparovich, Volha Harytskaya and Pavel Smrz. 2026. Tracking the evolution of LLM capabilities for Belarusian with OpenAI Evals. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.33

@inproceedings{poritski2026tracking,
  title = {{Tracking the evolution of LLM capabilities for Belarusian with OpenAI Evals}},
  author = {Vladislav Poritski, Oksana Volchek, Maksim Aparovich, Volha Harytskaya and Pavel Smrz},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.33},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.33}
}

References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evaluation of generative AI. Preprint, arXiv:2303.12528.
  2. Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-yeol Ahn. 2023. Can we trust the evaluation on Chat-GPT? In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pages 47–54, Toronto, Canada. Association for Computational Linguistics. find
  3. Maksim Aparovich, Volha Harytskaya, Vladislav Poritski, Oksana Volchek, and Pavel Smrz. 2025. BelarusianGLUE: Towards a natural language understanding benchmark for Belarusian. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5… find
  4. Thomas Ball, Shuo Chen, and Cormac Herley. 2024. Can we count on LLMs? The fixed-effect fallacy and claims of GPT-4 capabilities. Preprint, arXiv:2409.07638. — [arXiv:2409.07638](https://arxiv.org/abs/2409.07638) find
  5. Kathleen Bardovi-Harlig and Rex A. Sprouse. 2018. Negative versus positive transfer. The TESOL Encyclopedia of English Language Teaching, pages 1–6. find
  6. BIG-bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.
  7. Tsz Chung Cheng, Chung Shing Cheng, Chaak Ming Lau, Eugene Tin-Ho Lam, Chun Yat Wong, Hoi On Yu, and Cheuk Hei Chong. 2025. HKCanto-Eval: A benchmark for evaluating cantonese language understanding and cultural comprehension in llms. Preprint, arXiv:2503.12440. find
  8. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating LLMs by human preference. Preprint, arXiv:2403.04132.
  9. Hienadź Cychun. 2002. Weißrussisch. In Miloš Okuka and Gerald Krenn, editors, Lexikon der Sprachen des europäischen Ostens, pages 563–579. Wieser Verlag, Klagenfurt. find
  10. Chunyuan Deng, Yilun Zhao, Yuzhao Heng, Yitong Li, Jiannan Cao, Xiangru Tang, and Arman Cohan. 2024. Unveiling the spectrum of data contamination in language model: A survey from detection to remediation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16078–16092, Bang… find
  11. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  12. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  13. Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, and Arianna Bisazza. 2025. MultiBLiMP 1.0: A massively multilingual benchmark of linguistic minimal pairs. Preprint, arXiv:2504.02768. find
  14. Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yusofi, and Jörg Tiedemann. 2025. GlotEval: A test suite for massively multilingual evaluation of large language models… find
  15. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. Preprint, arXiv:1301.3781.
  16. OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, and 107 others. 2025. gpt-oss-120b & gpt-oss-2…
  17. Aske Plaat, Max van Duijn, Niki van Stein, Mike Preuss, Peter van der Putten, and Kees Joost Batenburg. 2025. Agentic large language models, a survey. Preprint, arXiv:2503.23037. — [arXiv:2503.23037](https://arxiv.org/abs/2503.23037)
  18. Anna Berenika Siwirska. 2024. Czy we współczesnej białoruszczyźnie istniej a dwie normy j˛ezykowe? Studia Białorutenistyczne, 18:269–283. find
  19. Ashima Suvarna, Harshita Khandelwal, and Nanyun Peng. 2024. PhonologyBench: Evaluating phonological skills of large language models. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 1–14, Bangkok, Thailand. Association for Computational Linguistics. find
  20. Matej Ulčar, Kristiina Vaik, Jessica Lindström, Milda Dailidėnaitė, and Marko Robnik-Šikonja. 2020. Multilingual culture-independent word analogy datasets. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4074–4080, Marseille, France. European Language Resources A… find
  21. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: A benchmark of linguistic minimal pairs for English. In Proceedings of the Society for Computation in Linguistics 2020, pages 409–410, New York, New York. Association for Computati… find
  22. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/poritski-2026-tracking-evolution-llm-capabilities-belarusian/