Elamur Research⌕
☰
home / papers / 2026 / Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER)

Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER)

Paper · 2026 · Felix Akeret

Summary

The paper addresses speech recognition for Swiss German, where a model hears dialect audio but must write Standard German, so it has to recognize and translate in one pass. The author fine-tunes Whisper large-v3 on up to 1,367 hours of broadcast audio paired with professionally written Standard German subtitles from SRF, the Swiss Parliament Corpus, institutional YouTube channels and PlaySuisse. Across 16 runs on a single DGX Spark, the study compares LoRA with full fine-tuning. On ASGDTS, which is kept fully separate from training, the best model scores 25.6% WER. When stylistic differences such as tense, word order and Swiss spelling are separated from real mistakes, the content WER drops to 13.8%, and a bias-corrected estimate gives 8.5%. The paper also argues that earlier reported results of about 17% WER are inflated. Plain Whisper trained on the ASGDTS test set itself, with no dialect data, reaches 13.88%. The work matters because it provides openly licensed, reproducible models and shows that the benchmark mostly rewards matching transcription conventions rather than understanding dialect.

Key points

  • Two models are released under Apache 2.0. The first is flix-swissgerman-lora (r=160, α=32), with 25.32% WER and 13.9% cWER. The second is flix-swissgerman-full (1,543M parameters), with 25.60% WER and 13.8% cWER.
  • The paper identifies LoRA alpha scaling as the cause of decoder hallucination. The popular α = 2r setting damages Whisper's end-of-sequence behavior; the tested ranges were r=32–200 and α/r=0.08–2.0.
  • Contamination test: vanilla Whisper self-trained on the ASGDTS test set reaches 13.88% WER, and Phi-4-multimodal reaches 3.9%. Models with prior dialect training gain less from this self-training.
  • Data source effects can be opposite even within one platform. PlaySuisse dialect films improved WER by 2.28 pp, while scripted series with noisy labels worsened it by 2.92 pp.
  • Evaluation uses a 200-sample ASGDTS subset (seed 42) for comparing runs and the full 5,750-sample set for final numbers. The total compute was about 960 GPU-hours.

Interesting ideas

  • I1. To see how much of a benchmark score comes from memorizing its conventions, train a model with no real domain knowledge directly on the test material; if it beats published systems, the benchmark measures format more than skill.
  • I2. Split the error metric into content errors and acceptable stylistic variants, because a single edit-distance score penalizes valid rephrasings as heavily as real failures.
  • I3. Weak labels from professionally written parallel text can replace restricted datasets, provided each source is checked and ablated on its own, since sources from the same platform can help or hurt.
  • I4. A fine-tuning heuristic that works in one architecture family can silently break another, and the damage showed up as a specific behavioral failure (not stopping) rather than as a smooth loss change.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Add this as a negative control in evaluation by tuning a model with no linguistic signal on the target treebank's annotation conventions. Gains below that ceiling should not count as real progress.
  • I2 (back): Score the text → graph → text compressor and translation separately for meaning-changing errors and for harmless changes such as tense or word order. Round-trip quality then reflects lost meaning, not surface drift.
  • I3 (learn): Track every silver-data source separately (witness parsers, LLM teacher tiers, corpus origin) and measure its marginal effect. Sources that consistently degrade results can then be dropped.
  • I4 (tree): When adapting gparse with adapters or new scaling settings, monitor targeted structural failures such as missing roots or broken sentence boundaries. Do not rely on aggregate LAS alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksSpeech & spoken language · 480 works

Cite

Cite as Felix Akeret. 2026. Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER). arXiv:2606.07608.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2606.07608 · arXiv 2606.07608

@misc{akeret2026subtitle,
  title = {{Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6\% WER (13.8\% cWER)}},
  author = {Felix Akeret},
  year = {2026},
  doi = {10.48550/arxiv.2606.07608},
  eprint = {2606.07608},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.07608}
}

References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abouelenin et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-LoRAs. arXiv:2503.01743, 2025. 5.6B multimodal model with mixture-of-LoRAs for vision and speech. ↗
  2. Verena Blaschke, Miriam Winkler, and Barbara Plank. Standard-to-dialect transfer trends differ across text and speech: A case study on intent and topic classification in german dialects. arXiv:2510.07890, 2025. ↗
  3. Nan Chen, Soledad Villar, and Soufiane Hayou. Learning rate scaling across LoRA ranks and transfer to full finetuning. arXiv:2602.06204, 2026. ↗
  4. Flavio D’Intino and Hans-Peter Hutter. Advancing STT for low-resource real-world speech. In HCI International 2025, volume 15822 of LNCS, pages 290–309. find
  5. Eyal Liron Dolev, Clemens Fidel Lutz, and Noëmi Aepli. Does whisper understand swiss german? an automatic, qualitative, and human evaluation. In Proc. VarDial Workshop, 2024. ↗
  6. Flurin17. whisper-large-v3-peft-swiss-german. HuggingFace Hub: Flurin17/whisper-large-v3-peft-swiss-german, 2024. LoRA/PEFT adapter on openai/whisper-large-v3. find
  7. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
  8. Daekeun Lee. Phi-4-multimodal fine-tuning for korean speech-to-text. 2025. 35K Korean speech samples. LoRA-only: 7.19% WER; encoder unfreeze: 3.54% WER on zeroth-test. find
  9. Yunpeng Liu, Xukui Yang, and Dan Qu. Exploration of whisper fine-tuning strategies for low-resource ASR. EURASIP Journal on Audio, Speech, and Music Processing, 2024(1):29, 2024. ↗
  10. Nizar Michaud. whisper-large-v3-turbo-swissgerman. HuggingFace Hub: nizarmichaud/whisper-large-v3-turbo-swissgerman, 2024. QLoRA, 17.5% WER on ASGDTS. find
  11. Michel Plüss, Lukas Neukom, and Manfred Vogel. SwissText 2021 Task 3: Swiss German Speech to Standard German Text. In Proceedings of the Swiss Text Analytics Conference, 2021. Defines the ASGDTS (All Swiss German Dialects Test Set); public validation split used as standard evaluation benchmark. find
  12. Michel Plüss, Manuela Hürlimann, Marc Cuny, Alla Stöckli, Nikolaos Kapotis, Julia Hartmann, Malgorzata Anna Ulasik, Christian Scheller, Yanick Schraner, Amit Jain, Jan Deriu, Mark Cieliebak, and Manfred Vogel. SDS-200: A swiss german speech to standard german text corpus. In Proc. LREC, 2022. find
  13. Michel Plüss, Jan Deriu, Yanick Schraner, Claudio Paonessa, Julia Hartmann, Larissa Schmidt, Christian Scheller, Manuela Hürlimann, Tanja Samardžić, Manfred Vogel, and Mark Cieliebak. STT4SG-350: A speech corpus for all swiss german dialect regions. In Proc. ACL (Short Papers), 2023. ↗
  14. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML, 2023. ↗
  15. Sebastian Raschka. Practical tips for finetuning LLMs using LoRA. Ahead of AI Newsletter, 2023. find
  16. Elizabeth Salesky, Matthew Wiesner, Jacob Bremerman, Roldano Cattoni, Matteo Negri, Marco Turchi, Douglas W Oard, and Matt Post. The multilingual TEDx corpus for speech recognition and translation. In Proc. Interspeech, 2021. find
  17. Zitha Sasindran, Harsha Yelchuri, T. V. Prabhakar, and Supreeth Rao. H eval : A new hybrid evaluation metric for automatic speech recognition tasks. arXiv:2211.01722, 2022. ↗
  18. Yanick Schraner, Christian Scheller, Michel Plüss, and Manfred Vogel. Swiss german speech to text system evaluation. arXiv:2207.00412, 2022. ↗
  19. Clément Sicard, Victor Gillioz, and Kajetan Pyszkowski. Spaiche: Extending state-of-the-art ASR models to swiss german dialects. In Proc. SwissText, 2023. ↗
  20. Zheshu Song, Jianheng Zhuo, Yifan Yang, Ziyang Ma, Shixiong Zhang, and Xie Chen. LoRA-Whisper: Parameter-efficient and extensible multilingual ASR. In Proc. Interspeech, 2024. find
  21. Vincenzo Timmel, Claudio Paonessa, Reza Kakooee, Manfred Vogel, and Daniel Perruchoud. Fine-tuning whisper on low-resource languages for real-world applications. arXiv:2412.15726, 2024. ↗
  22. Yuan Tseng, Titouan Parcollet, Rogier van Dalen, Shucong Zhang, and Sourav Bhattacharya. Evaluation of LLMs in speech is often flawed: Test set contamination in large language models for speech recognition. arXiv:2505.22251, 2025. ↗
  23. Yingzhi Wang, Anas Alhmoud, Saad Alsahly, Muhammad Alqurishi, and Mirco Ravanelli. Calm-whisper: Reduce whisper hallucination on non-speech by calming crazy heads down. arXiv:2505.12969, 2025. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/akeret-2026-subtitle-aligned-fine-tuning-whisper/