Elamur Research⌕
☰
home / papers / 2025 / A Multi-Dialectal Dataset for German Dialect ASR and Dialect-to-Standard Speech Translation

A Multi-Dialectal Dataset for German Dialect ASR and Dialect-to-Standard Speech Translation

Paper · 2025 · Verena Blaschke, Miriam Winkler, Constantin Forster et al.

Summary

German dialects are rarely covered in speech recognition research. Their gap from Standard German also goes beyond pronunciation: vocabulary, morphology and syntax differ too. The authors build Betthupferl, an evaluation set of read children's bedtime stories broadcast by Bayerischer Rundfunk. It has about four hours of dialect speech from all seven administrative regions of Bavaria, covering the Franconian, Bavarian and Alemannic (Swabian) groups, plus half an hour of Standard German. Every dialect sentence has two references: an ad-hoc spelling of the dialect and a Standard German translation. A word-level annotation of a subset sorts the differences between the two references into categories. Several multilingual ASR families are then benchmarked: Whisper (original, German-tuned and Swiss-German-tuned), XLS-R and MMS. They are scored with WER, CER and BLEU, and the metrics are checked against human judgments. The best system is Whisper large-v3. Its output sometimes standardizes dialect grammar, but it often keeps dialect constructions, some acceptable and some wrong. This is the first ASR test resource for these dialects, and it makes visible which "errors" are really legitimate variation.

Key points

  • The dataset has 3,788 sentences (about 273 min) from 22 speakers. The dialect portion is 3,257 sentences (241 min), with 32–37 min per Bavarian region and a mean utterance length of 4.3 s.
  • On average, the dialect and Standard German references differ by 57% normalized word-level Levenshtein distance and 24% at character level.
  • The annotated subset has 305 sentences (about 4.2k words) and nine difference categories:
    • about 45% of words are identical across the two references;
    • about 47% differ in phonetic or morphological spelling;
    • word splitting affects only 4% of words but appears in 41% of sentences.
  • For Whisper large-v3, behaviour depends on the difference category:
    • a determiner before a name is mostly kept in a valid informal way (77%);
    • word-choice and verb-phrase differences end up as errors 63% of the time;
    • dropped or fused pronouns end up as errors 60% of the time.
  • Transcriptions, annotations, guidelines, code and predictions are released at github.com/mainlp/betthupferl. The audio is available from Bayerischer Rundfunk on request.

Interesting ideas

  • I1. Each output word is judged against two references and labelled as matching the standard, deviating but acceptable, or wrong, so legitimate variation is not counted as failure.
  • I2. The distance between the two references is drawn as a baseline line in the error-rate plots, which shows how much of a model's error is simply the inherent dialect–standard gap.
  • I3. Differences are annotated with a small fixed taxonomy (determiner plus name, possessive construction, dropped pronoun, word order, word splitting), which turns a vague notion of "dialect difference" into countable phenomena.
  • I4. The varieties are not parallel, but they are held to one genre and style (short children's stories), so the comparison across varieties is controlled without needing translations of the same text.
  • I5. Automatic metrics are validated against human judgments before the scores are trusted.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova's evaluations could add an "acceptable variant" class, so that parser or answer outputs differing from gold in a licensed way are separated from real errors.
  • I2 (learn): when scoring against a converted or alternative-dialect treebank, Mova could report the gold-to-gold divergence as a floor for interpreting LAS drops.
  • I3 (tree): a similar phenomenon inventory could drive and audit UD dialect-conversion rules, counting how often each construction appears and breaks the parser.
  • I4 (answer): fable and tale QA test sets could be built as genre-matched but non-parallel splits, to probe robustness to style or variety shifts.
  • I5 (learn): silver-data agreement scores and LLM-teacher grades could be spot-checked against human ratings, to confirm the metric tracks real quality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 480 works

Cite

Cite as Verena Blaschke, Miriam Winkler, Constantin Forster, Gabriele Wenger-Glemser and Barbara Plank. 2025. A Multi-Dialectal Dataset for German Dialect ASR and Dialect-to-Standard Speech Translation. arXiv:2506.02894.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.21437/interspeech.2025-318 · arXiv 2506.02894

@misc{blaschke2025multi,
  title = {{A Multi-Dialectal Dataset for German Dialect ASR and Dialect-to-Standard Speech Translation}},
  author = {Verena Blaschke, Miriam Winkler, Constantin Forster, Gabriele Wenger-Glemser and Barbara Plank},
  year = {2025},
  doi = {10.21437/interspeech.2025-318},
  eprint = {2506.02894},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.21437/interspeech.2025-318}
}

References

34 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Tatman and C. Kasten, “Effects of talker dialect, gender & race on accuracy of Bing Speech and YouTube automatic captions,” in Interspeech 2017, 2017, pp. 934–938. find
  2. N. Markl, “Language variation and algorithmic bias: understanding algorithmic bias in British English automatic speech recognition,” in Proc. FAccT, 2022, p. 521–534. find
  3. V. Blaschke, C. Purschke, H. Schütze, and B. Plank, “What do dialect speakers want? A survey of attitudes towards language technology for German dialects,” in Proc. ACL, 2024, pp. 823– 841. find
  4. A. Aksënova, Z. Chen, C.-C. Chiu, D. van Esch, P. Golik, W. Han, L. King, B. Ramabhadran, A. Rosenberg, S. Schwartz, and G. Wang, “Accented speech recognition: Benchmarking, pre-training, and diverse data,” 2022, preprint on arXiv 2205.08014. ↗
  5. P. Wiesinger, “Die Einteilung der deutschen Dialekte,” in Dialektologie: Ein Handbuch zur deutschen und allgemeinen Dialektforschung, 1983, pp. 807–900. find
  6. O. Ahia, A. Aremu, D. Abagyan, H. Gonen, D. I. Adelani, D. Abolade, N. A. Smith, and Y. Tsvetkov, “Voices unheard: NLP resources and models for Yorùbá regional dialects,” in Proc. EMNLP, 2024, pp. 4392–4409. ↗
  7. N. Markl, E. Wallington, O. Klejch, T. Reitmaier, G. Bailey, J. Pearson, M. Jones, S. Robinson, and P. Bell, “Automatic transcription and (de)standardisation,” in Proc. SIGUL, 2023, pp. 93– 97. find
  8. I. Nigmatulina, T. Kew, and T. Samardzic, “ASR for non-standardised languages with dialectal variation: the case of Swiss German,” in Proc. Workshop on NLP for Similar Languages, Varieties and Dialects, 2020, pp. 15–24. find
  9. V. Blaschke, H. Schütze, and B. Plank, “A survey of corpora for Germanic low-resource languages and dialects,” in Proc. NoDaLiDa, 2023, pp. 392–414. ↗
  10. Y. Scherrer, T. Samardžić, and E. Glaser, “ArchiMob: Ein multidi-alektales Korpus schweizerdeutscher Spontansprache,” Linguistik Online, vol. 98, no. 5, 2019. find
  11. P. Dogan-Schönberger, J. Mäder, and T. Hofmann, “SwissDial: Parallel multidialectal corpus of spoken Swiss German,” 2021, preprint on arXiv 2103.11401. ↗
  12. R. Herms, L. Seelig, S. Münch, and M. Eibl, “A corpus of read and spontaneous Upper Saxon German speech for ASR evaluation,” in Proc. LREC, 2016, pp. 4648–4651. find
  13. M. Plüss, M. Hürlimann, M. Cuny, A. Stöckli, N. Kapotis, J. Hartmann, M. A. Ulasik, C. Scheller, Y. Schraner, A. Jain, J. Deriu, M. Cieliebak, and M. Vogel, “SDS-200: A Swiss German speech to Standard German text corpus,” in Proc. LREC, 2022, pp. 3250– 3256. find
  14. M. Plüss, J. Deriu, Y. Schraner, C. Paonessa, J. Hartmann, L. Schmidt, C. Scheller, M. Hürlimann, T. Samardžić, M. Vogel, and M. Cieliebak, “STT4SG-350: A speech corpus for all Swiss German dialect regions,” in Proc. ACL, 2023, pp. 1763–1772. ↗
  15. E. Dolev, C. Lutz, and N. Aepli, “Does Whisper understand Swiss German? An automatic, qualitative, and human evaluation,” in Proc. VarDial, 2024, pp. 28–40. ↗
  16. C. Sicard, V. Gillioz, and K. Pyszkowski, “Spaiche: Extending state-of-the-art ASR models to Swiss German dialects,” in Proc. SwissText, 2023, pp. 76–83. ↗
  17. J. Gerlach, J. Mutal, and P. Bouillon, “Producing Standard German subtitles for Swiss German TV content,” in Proc. SLPAT, 2022, pp. 37–43. find
  18. T. Schmidt and W. Schütte, “FOLKER: An annotation tool for efficient transcription of natural, multi-party interaction,” in Proc. LREC, 2010. find
  19. E. Artemova, V. Blaschke, and B. Plank, “Exploring the robustness of task-oriented dialogue systems for colloquial German varieties,” in Proc. EACL, 2024, pp. 445–468. find
  20. V. Blaschke, B. Kovačić, S. Peng, H. Schütze, and B. Plank, “MaiBaam: A multi-dialectal Bavarian Universal Dependency treebank,” in Proc. LREC-COLING, 2024, pp. 10 921–10 938. find
  21. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proc. ICML, 2023. ↗
  22. M. Schubert, D. Duran, and I. Siegert, “Challenges of German speech recognition: A study on multi-ethnolectal speech among adolescents,” Interspeech 2024, pp. 3045–3049, 2024. find
  23. J. Gorisch and T. Schmidt, “Evaluating workflows for creating orthographic transcripts for oral corpora by transcribing from scratch or correcting ASR-output,” in Proc. LREC-COLING, 2024, pp. 6564–6574. find
  24. C. Harris, C. Mgbahurike, N. Kumar, and D. Yang, “Modeling gender and dialect bias in automatic speech recognition,” in Findings of the ACL: EMNLP 2024, 2024, pp. 15 166–15 184.
  25. A. Kantharuban, I. Vulić, and A. Korhonen, “Quantifying the dialect gap and its correlates across languages,” in Findings of the ACL: EMNLP 2023, 2023, pp. 7226–7245. find
  26. R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common Voice: A massively-multilingual speech corpus,” in Proc. LREC, 2020, pp. 4218–4222. ↗
  27. B. Huang, “German-optimized Whisper models for speech recognition,” 2023, accessed on 17.02.2025, huggingface.co/collections/bofenghuang/german-whisperv00-67332b06c3b181f9acbd8130. find
  28. N. Michaud, “Whisper large v3 turbo fine-tuned for Swiss German,” 2024, accessed on 17.02.2025, huggingface.co/nizarmichaud/whisper-large-v3-turbo-swissgerman. find
  29. A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “XLS-R: Self-supervised cross-lingual speech representation learning at scale,” in Interspeech 2022, 2022, pp. 2278–2282.
  30. A. McDowell, “Wav2vec2-xls-r-300m fine-tuned on CommonVoice7 - DE,” 2022, accessed on 17.02.2025, huggingface.co/AndrewMcDowell/wav2vec2-xls-r-300m-german-de. find
  31. ——, “Wav2vec2-xls-r-1b fine-tuned on Common-Voice8 - DE,” 2022, accessed on 17.02.2025, huggingface.co/AndrewMcDowell/wav2vec2-xls-r-1B-german. find
  32. V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi, A. Baevski, Y. Adi, X. Zhang, W.-N. Hsu, A. Conneau, and M. Auli, “Scaling speech technology to 1,000+ languages,” Journal of Machine Learning Research, vol. 25, no. 97, pp. 1–52, 2024. ↗
  33. M. Post, “A call for clarity in reporting BLEU scores,” in Proc. WMT, 2018, pp. 186–191.
  34. N. Aepli, C. Amrhein, F. Schottmann, and R. Sennrich, “A benchmark for evaluating machine translation metrics on dialects without standard orthography,” in Proc. WMT, 2023, pp. 1045–1065. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/blaschke-2025-multi-dialectal-dataset-german-dialect/