Elamur Research⌕
☰
home / papers / 2026 / State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting?

State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting?

Paper · 2026 · Taja Kuzman Pungeršek, Peter Rupnik, Ivan Porupski et al.

Summary

The paper asks whether zero-shot prompting of instruction-tuned LLMs has overtaken fine-tuned BERT-like encoders as the best-performing approach to classifying texts written in South Slavic languages, which have received little evaluation so far. The authors benchmark openly available fine-tuned classifiers (XLM-R-ParlaSent, X-GENRE, IPTC News Topic classifier, ParlaCAP classifier) against open-weight and closed-source LLMs. They use four benchmark families covering three tasks and three domains: ParlaSent (sentiment in parliamentary speeches), GINCO (web genres), IPTC (news topics) and ParlaCAP (parliamentary policy topics). The languages are Slovenian, Croatian, Serbian, Bosnian and Macedonian, with English for comparison. Models are scored with micro-F1 and macro-F1, and a majority-class dummy classifier serves as a baseline.

Zero-shot LLMs frequently equal or beat the fine-tuned encoders, and their scores on South Slavic data are roughly as good as on English. However, LLMs produce less predictable outputs, run much more slowly and cost more to use. For annotating large corpora, fine-tuned BERT-like models therefore remain the more practical option. This is a useful guide for less-resourced language projects.

Key points

  • Test sets vary widely in size and label granularity:
    • ParlaSent: 3 labels, from 190 (BS) to 2600 (EN) instances.
    • GINCO: 8 genres, 80–272 texts.
    • IPTC: 17 top-level IPTC NewsCodes topics, about 300 texts per language.
    • ParlaCAP: 22 CAP Master Codebook labels, about 820–880 speeches per language.
  • The evaluated LLMs were:
    • Closed-source: GPT-3.5-Turbo, GPT-4o, GPT-5, Gemini 2.5 Flash, Mistral Medium 3.1.
    • Open-weight: LLaMA 3.3 (70B), Gemma 3 (27B), DeepSeek-R1-Distill-Qwen-14B, Qwen 3 (32B).
  • Open-weight models ran locally through Ollama in quantized form. All LLMs received identical prompts and default settings, except temperature, which was set to 0.
  • Annotation reliability was checked with Krippendorff's alpha, with values near or above 0.667. To avoid training-data contamination, only ParlaSent is public; the other test sets are available on request.
  • The evaluation code is released on GitHub (TajaKuzman/Benchmarking-Text-Classification-on-South-Slavic). The classifiers are available on Hugging Face (classla) and the training data on CLARIN.SI.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of LjubljanaLjubljana · Slovenia13 works · 38,693 citations🔬Institute of Contemporary HistoryLjubljana · Slovenia4 works · 32 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Taja Kuzman Pungeršek, Peter Rupnik, Ivan Porupski, Vuk Dinić and Nikola Ljubešić. 2026. State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting?. Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2fudy99w2taz

@inproceedings{pungersek2026state,
  title = {{State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting?}},
  author = {Taja Kuzman Pungeršek, Peter Rupnik, Ivan Porupski, Vuk Dinić and Nikola Ljubešić},
  booktitle = {Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026},
  year = {2026},
  doi = {10.63317/2fudy99w2taz},
  url = {https://doi.org/10.63317/2fudy99w2taz}
}

References

55 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948.
  2. Marta Bañón, Miquel Esplà-Gomis, Mikel L Forcada, Cristian García-Romero, Taja Kuzman, Nikola Ljubešić, Rik van Noord, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Peter Rupnik, et al. 2022. MaCoCu: Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages.… find
  3. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. arXiv preprint arXiv:2302.09210. find
  4. Frank R Baumgartner, Christian Breunig, and Emiliano Grossman. 2019. Comparative Policy Agendas: Theory, Tools, Data. Oxford University Press. find
  5. Fan Huang, Haewoon Kwak, and Jisun An. 2023. Is ChatGPT better than human annotators? Potential and limitations of ChatGPT in explaining implicit hate speech. In Companion Proceedings of the ACM Web Conference 2023, pages 294–297. find
  6. Shaun Bevan. 2019. Gone Fishing. Comparative Policy Agendas: Theory, Tools, Data, pages 17– 34. find
  7. IPTC. 2022. Groups of NewsCodes. https://iptc. org/standards/newscodes/groups/\#descrncd. Accessed October 29, 2024. find
  8. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv prep…
  9. Miloš Jakubíček, Adam Kilgarriff, Vojtěch Kovář, Pavel Rychlỳ, and Vít Suchomel. 2013. The Ten-Ten corpus family. In 7th international corpus linguistics conference CL, pages 125–127. Lancaster University. find
  10. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In 58th Annual Meeting of the Association for Computational Lin… find
  11. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT, pages 4171–4186.
  12. Arina Kostina, Marios D Dikaiakos, Dimosthenis Stefanidis, and George Pallis. 2025. Large language models for text classification: Case study and comprehensive review. arXiv preprint arXiv:2501.08457. — [arXiv:2501.08457](https://arxiv.org/abs/2501.08457) find
  13. Orphée De Clercq, Luna De Bruyne, and Véronique Hoste. 2020. News topic classification as a first step towards diverse news recommendation. Computational Linguistics in the Netherlands Journal, 10:37–55. find
  14. Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage Publications. find
  15. Tomaž Erjavec, Matyáš Kopp, Nikola Ljubešić, Taja Kuzman, Paul Rayson, Petya Osenova, Maciej Ogrodniczuk, Çağrı Çöltekin, Danijel Koržinek, Katja Meden, et al. 2025. ParlaMint II: advancing comparable parliamentary corpora across Europe. Language Resources and Evaluation, 59(3):2071–2102. find
  16. Taja Kuzman and Nikola Ljubešić. 2023. Automatic genre identification: a survey. Language Resources and Evaluation, pages 1–34. find
  17. Taja Kuzman and Nikola Ljubešić. 2025. LLM Teacher-Student Framework for Text Classification With No Manually Annotated Data: A Case Study in IPTC News Topic Classification. IEEE Access, 13:35621–35633. find
  18. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  19. Taja Kuzman, Igor Mozetič, and Nikola Ljubešić. 2023. Automatic Genre Identification for Robust Enrichment of Massive Text Collections: Investigation of Classification Methods in the Era of Large Language Models. Machine Learning and Knowledge Extraction, 5(3):1149–1175. find
  20. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Taja Kuzman, Peter Rupnik, and Nikola Ljubešić. 2022. The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild. In Language Resources and Evaluation Conference, pages 1584–1594, Marseill… find
  21. Michal Mochtak, Peter Rupnik, Taja Kuzman, and Nikola Ljubešić. 2025. Parlasent: mapping sentiment in political discourse with large language models. Political Research Exchange, 7(1):2508377. find
  22. Michal Mochtak, Peter Rupnik, and Nikola Ljubešić. 2024. The ParlaSent Multilingual Training Dataset for Sentiment Identification in Parliamentary Proceedings. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)… find
  23. Taja Kuzman Pungeršek, Peter Rupnik, Daniela Širinić, and Nikola Ljubešić. 2026. Supercharging Agenda Setting Research: The ParlaCAP Dataset of 28 European Parliaments and a Scalable Multilingual LLM-Based Classification. arXiv preprint arXiv:2602.16516.
  24. OpenAI. 2023. ChatGPT General FAQ. https://help.openai.com/en/articles/ 6783457-chatgpt-general-faq. Accessed: June 26, 2025. ↗
  25. Nikola Ljubešić, Nada Galant, Sonja Benčina, Jaka Čibej, Stefan Milosavljević, Peter Rupnik, and Taja Kuzman. 2024a. DIALECT-COPA: Extending the Standard Translations of the COPA Causal Commonsense Reasoning Dataset to South Slavic Dialects. In Proceedings of the Eleventh Workshop on NLP for Similar… find
  26. OpenAI. 2024. Hello GPT-4o. https://openai.com/ index/hello-gpt-4o/. Accessed September 11, 2024. ↗
  27. OpenAI. 2025. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/. Accessed: October 10, 2025. ↗
  28. Nikola Ljubešić, Taja Kuzman, Peter Rupnik, Ivan Vulić, Fabian Schmidt, and Goran Glavaš. 2024b. JSI and WüNLP at the DIALECT-COPA Shared Task: In-Context Learning From Just a Few Dialectal Examples Gets You Quite Far. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties,… find
  29. Wanda J Orlikowski and JoAnne Yates. 1994. Genre repertoire: The structuring of communicative practices in organizations. Administrative science quarterly, pages 541–574. find
  30. Alina Petukhova and Nuno Fachada. 2023. MN-DS: A multilabeled news dataset for news articles hierarchical classification. Data, 8(5):74. find
  31. Nikola Marić, Boshko Koloski, Damjan Demšar, Jan Jona Javoršek, and Sašo Džeroski. 2025. Running large language models locally: design and operational insights with llm.ijs.si. International conference AI for science 2025: Ljubljana, Slovenia, 22.09.2025-26.09.2025, page 77. find
  32. Qwen Team. 2024a. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  33. Qwen Team. 2024b. Qwen2.5: A party of foundation models. Accessed: June 26, 2025. find
  34. Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023. Text Classification via Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8990–9005.
  35. Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating training data with language models: Towards zero-shot language understanding. Advances in Neural Information Processing Systems, 35:462–477.
  36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  37. Meta. 2024. Llama 3.3 Model Card. https://github.com/meta-llama/llama-models/blob/ main/models/llama3_3/MODEL_CARD.md. Accessed: June 26, 2025. ↗
  38. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  39. Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2020. Deep Learning Based Text Classification: A Comprehensive Review. arXiv preprint arXiv:2004.03705. find
  40. Bowen Zhang, Daijun Ding, Liwen Jing, Genan Dai, and Nan Yin. 2022. How would Stance Detection Techniques Evolve after the Launch of ChatGPT? arXiv preprint arXiv:2212.14548. find
  41. Mistral AI. 2025. Medium is the new large. https://mistral.ai/news/mistral-medium-3. Accessed: October 10, 2025. ↗
  42. Yazhou Zhang, Mengyao Wang, Qiuchi Li, Prayag Tiwari, and Jing Qin. 2025. Pushing the limit of LLM capacity for text classification. In Companion Proceedings of the ACM on Web Conference 2025, pages 1524–1528. find
  43. Kuzman Pungeršek, Taja and Ljubešić, Nikola. 2026. Multilingual training dataset for CAP policy topic classification ParlaCAP-train. Slovenian language resource repository CLARIN.SI. PID http://hdl.handle.net/11356/2093. ↗
  44. Hang Zhao, Qile P Chen, Yijing Barry Zhang, and Gang Yang. 2024. Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuning. arXiv preprint arXiv:2412.08587. — [arXiv:2412.08587](https://arxiv.org/abs/2412.08587) find
  45. Ljubešić, Nikola and Rupnik, Peter and van Noord, Rik. 2023. Multilingual parliamentary model XLM-R-parla. Hugging Face. PID https://doi.org/10.57967/hf/6717.10. — [doi:10.57967/hf/6717.10](https://doi.org/10.57967/hf/6717.10) ↗
  46. Mochtak, Michal and Rupnik, Peter and Meden, Katja and Ljubešić, Nikola. 2023. The multilingual sentiment dataset of parliamentary debates ParlaSent 1.0. Slovenian language resource repository CLARIN.SI. PID http://hdl.handle.net/11356/1868. ↗
  47. Language Resource References Rupnik, Peter and Ljubešić, Nikola and Mochtak, Michal. 2023. Multilingual parliament sentiment regression model XLM-R-ParlaSent. Hugging Face. PID https://doi.org/10.57967/hf/6718. — [doi:10.57967/hf/6718](https://doi.org/10.57967/hf/6718) ↗
  48. Erjavec, Tomaž and Kopp, Matyáš and Ogrodniczuk, Maciej and Osenova, Petya and others. 2024. Multilingual comparable corpora of parliamentary debates ParlaMint 4.1. Slovenian language resource repository CLARIN.SI. PID http://hdl.handle.net/11356/1912. ↗
  49. Kuzman, Taja and Ljubešić, Nikola. 2023. Multilingual text genre classification model X-GENRE. Hugging Face. PID https://doi.org/10.57967/hf/0927. — [doi:10.57967/hf/0927](https://doi.org/10.57967/hf/0927) ↗
  50. Kuzman, Taja and Ljubešić, Nikola. 2024a. English-Slovenian text genre dataset X-GENRE. Slovenian language resource repository CLARIN.SI. PID http://hdl.handle.net/11356/1960. ↗
  51. Kuzman, Taja and Ljubešić, Nikola. 2024b. Genre-enriched web corpora MaCoCu-Genre. Slovenian Language Resource Repository CLARIN.SI. PID http://hdl.handle.net/11356/1969. ↗
  52. Kuzman, Taja and Ljubešić, Nikola. 2024c. Multilingual IPTC Media Topic dataset EMMediaTopic 1.0. Slovenian Language Resource Repository CLARIN.SI. PID http://hdl.handle.net/11356/1991. ↗
  53. Kuzman, Taja and Ljubešić, Nikola. 2024d. Multilingual text genre classification model X-GENRE. Slovenian language resource repository CLARIN.SI. PID http://hdl.handle.net/11356/1961. ↗
  54. Kuzman, Taja and Ljubešić, Nikola. 2025. Multilingual IPTC News Topic Classifier. Hugging Face. PID https://doi.org/10.57967/hf/4709. — [doi:10.57967/hf/4709](https://doi.org/10.57967/hf/4709) ↗
  55. Kuzman Pungeršek, Taja and Ljubešić, Nikola. 2025. Multilingual ParlaCAP model for CAP Topic Classification in Parliamentary Speeches. Hugging Face. PID https://doi.org/10.57967/hf/6684. — [doi:10.57967/hf/6684](https://doi.org/10.57967/hf/6684) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pungersek-2026-state-art-text-classification-south/