Elamur Research⌕
☰
home / papers / 2026 / Regional Variation in the Performance of ASR Models on Croatian and Serbian

Regional Variation in the Performance of ASR Models on Croatian and Serbian

Summary

The authors ask whether ASR for Croatian and Serbian, two closely related varieties of the BCMS macrolanguage, needs separate data for each language and region, or whether data from one can serve both. They evaluate seven freely available models on the Mak na konac (MNK) test set, which contains 20 hours of manually transcribed interview-style broadcast speech. The set is split evenly across northern and southern regions of each country. Because each segment has several reference transcripts, the best-matching one is used for scoring, which neutralises differences in transcription conventions. The models cover three setups:

  • zero-shot Whisper,
  • Whisper fine-tuned on Serbian,
  • NeMo and wav2vec2 models trained or fine-tuned on Croatian parliamentary speech.

Stronger models turn out to be fairly robust to regional variation. Systems trained from scratch on Croatian alone perform well on Serbian too. Fine-tuning large multilingual models on modest datasets underperforms both zero-shot use and training from scratch. Some performance differences appear to reflect speaking style rather than geography. The study offers practical guidance for building ASR in macrolanguages and other similar-language settings.

Key points

  • MNK test set: four 5-hour parts.

    • HR1: Zagreb, Radio Student
    • HR2: Split, TV Dalmacija
    • SR1: Belgrade, Peščanik
    • SR2: Niš, Južne vesti

    The audio is public on Hugging Face (classla/mak_na_konac). Transcripts are withheld to prevent contamination, but researchers can request evaluation reports for submitted output.

  • Models evaluated:

    • WhisperV (large-v3, zero-shot)
    • WhisperS (~70 h Serbian fine-tuning) and WhisperSJV (~120 h Serbian fine-tuning)
    • NeMo Transducer and CTC, trained from scratch on 1816 h of ParlaSpeech-HR
    • W2V2XLSR and W2V2Slavic, fine-tuned on 300 h of ParlaSpeech-HR

  • Fine-tuning: in these setups, fine-tuned models were consistently weaker than both zero-shot Whisper and the from-scratch models.

  • Score variability: the average number of distinct WER scores per data part was HR1 412, SR1 335, SR2 276 and HR2 266. Overall model performance follows the same ordering. The authors attribute this to fillers, repetitions, numbers and abbreviations, which reflect speech style rather than region.

  • Limitations: the training setups were not controlled. In particular, there is no Whisper model fine-tuned on Croatian and no model trained from scratch on Serbian or on mixed Croatian-Serbian data. The authors suggest that combining Croatian and Serbian training data is the most promising direction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.vardial-1.20

@inproceedings{samardzic2026regional,
  title = {{Regional Variation in the Performance of ASR Models on Croatian and Serbian}},
  author = {Samardžić, Tanja and Rupnik, Peter and Ljubešić, Nikola},
  booktitle = {Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects},
  year = {2026},
  doi = {10.18653/v1/2026.vardial-1.20},
  url = {https://doi.org/10.18653/v1/2026.vardial-1.20}
}

References

19 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maryam Khalifa Al Ali and Hanan Aldarmaki. 2024. Mixat: A data set of bilingual emirati-English speech. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024, pages 222–226, Torino, Italia. ELRA and ICCL. find
  2. Mohammad Al-Fetyani, Muhammad Al-Barham, Gheith Abandah, Adham Alsharkawi, and Maha Dawas. 2021. Masc: Massive arabic speech corpus. find
  3. Sadeen Alharbi, Areeb Alowisheq, Zoltán Tüske, Kareem Darwish, Abdullah Alrajeh, Abdulmajeed Al-rowithi, Aljawharah Bin Tamran, Asma Ibrahim, Nikola Ljubešić, Peter Rupnik, and Tea Perinčić. 2024c. The "Mići Princ" text and speech dataset of Chaka-vian micro-dialects. Slovenian language resource… find
  4. Hamdy Mubarak, Amir Hussein, Shammur Absar Chowdhury, and Ahmed Ali. 2021. QASR: QCRI aljazeera speech resource a large scale annotated Arabic speech corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu… find
  5. Iuliia Nigmatulina, Tannon Kew, and Tanja Samardzic. 2020. ASR for non-standardised languages with dialectal variation: the case of Swiss German. In Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 15–24, Barcelona, Spain (Online). International Committee o… find
  6. Michel Plüss, Jan Deriu, Yanick Schraner, Claudio Paonessa, Julia Hartmann, Larissa Schmidt, Christian Scheller, Manuela Hürlimann, Tanja Samardžić, Manfred Vogel, and Mark Cieliebak. 2023. STT4SG-350: A speech corpus for all Swiss German dialect regions. In Proceedings of the 61st Annual Meeting o… find
  7. Michel Plüss, Manuela Hürlimann, Marc Cuny, Alla Stöckli, Nikolaos Kapotis, Julia Hartmann, Malgorzata Anna Ulasik, Christian Scheller, Yanick Schraner, Amit Jain, Jan Deriu, Mark Cieliebak, and Manfred Vogel. 2022. SDS-200: A Swiss German speech to Standard German text corpus. In Proceedings of the… find
  8. Tanja Samardžić, Yves Scherrer, and Elvira Glaser. 2016. ArchiMob - a corpus of spoken Swiss German. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4061–4066, Portorož, Slovenia. European Language Resources Association (ELRA). find
  9. Tanja Samardžić, Peter Rupnik, Mirjana Starović, and Nikola Ljubešić. 2024. Mak na konac: A multi-reference speech-to-text benchmark for croatian and serbian. Institute of Contemporary History. find
  10. Yves Scherrer, Tanja Samardžić, and Elvira Glaser. 2019. Digitising Swiss German: how to process and study a polycentric spoken language. Language Resources & Evaluation, 53:735–769. find
  11. Yanick Schraner, Christian Scheller, Michel Plüss, and Manfred Vogel. 2022. Swiss german speech to text system evaluation. Preprint, arXiv:2207.00412. find
  12. Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Miguel Pino, and Emmanuel Dupoux. 2021. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. CoRR, abs/2101.00390. find our card
  13. Michel Plüss, Lukas Neukom, Christian Scheller, and Manfred Vogel. 2021. Swiss parliaments corpus, an automatically aligned swiss german speech to standard german text corpus. Preprint, arXiv:2010.02810. find
  14. Branislav Popović, Stevan Ostrogonac, Edvin Pakoci, Nikša Jakovljević, and Vlado Delić. 2015. Deep neural network based continuous speech recognition for serbian using the kaldi toolkit. In Speech and Computer, pages 186–192, Cham. Springer International Publishing. find
  15. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. 2011. The kaldi speech recognition toolkit. In IEEE 2011 Workshop on Automatic Speech Recognition and… find
  16. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR. find our card
  17. Peter Rupnik and Nikola Ljubešić. 2022a. ASR training dataset for serbian JuzneVesti-SR v1.0. Slovenian language resource repository CLARIN.SI. find
  18. Peter Rupnik and Nikola Ljubešić. 2022b. ASR training dataset for Serbian JuzneVesti-SR v1.0. Slovenian language resource repository CLARIN.SI. find
  19. Andrija Sagić. 2023. Whisper-large-v3-sr-combined. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/samardzic-2026-regional-variation-performance-asr-models/