Elamur Research⌕
☰
home / papers / 2025 / Evaluating Standard and Dialectal Frisian ASR: Multilingual Fine-tuning and Language Identification for Improved Low-resource Performance

Evaluating Standard and Dialectal Frisian ASR: Multilingual Fine-tuning and Language Identification for Improved Low-resource Performance

Paper · 2025 · Reihaneh Amooie, Wietse de Vries, Yun Hao et al.

Summary

The paper asks how to build better speech recognition for Frisian, a low-resource minority language of the Netherlands, and how well such systems handle its regional dialects (Clay, Wood and South Frisian). The authors fine-tune the self-supervised XLS-R 1B model on Common Voice 17.0 Frisian. They then add equal-sized Common Voice portions of Dutch, German and English, one language at a time, ordered by lexical-phonetic closeness to Frisian. They also test prepending a language-identification (LID) token to each target transcription. For dialect evaluation they recorded new speech with the SPRAAKLAB mobile lab. Older speakers read prompts either in Dutch, which they translated mentally, or in Standard Frisian, and spoke each sentence in their own variety. Adding Dutch and German lowers the word error rate, English hurts, and LID gives a small extra gain. Dialectal speech is recognised clearly worse than standard read speech, and Dutch prompts elicit more dialectal speech. The main lesson is that test sets built only from standard-language prompts can paint too rosy a picture of real-world accuracy.

Key points

  • Baseline fine-tuning data is about 5.5 h of Common Voice 17.0 Frisian (3,921 sentences, 195 speakers). The multilingual sets grow to 11 h (fy-nl), 16 h (fy-nl-de) and 22 h (fy-nl-de-en). Each added language is downsampled to the Frisian size.
  • The model is XLS-R 1B: 48 Transformer encoder layers, pre-trained on 436k hours in 128 languages. It was fine-tuned with CTC loss on one A100 with the feature extractor frozen.
  • Language order follows ASJP-based LDND distances: Dutch first, then German, then English. Frisian+Dutch+German cut WER by 1.1 points absolute (about 7.5% relative) over the monolingual baseline. Adding English pushed WER back up to 13.4%.
  • The SPRAAKLAB dialect set has 16 men over 60 and two subsets of 824 sentences each (103 per speaker). One subset used Dutch prompts and the other Standard Frisian prompts, with regions balanced across the two.
  • On dialect data, WERs sit roughly in the 24–27% range. The Dutch/German benefit holds there only when LID is used. LID recall stays high but drops from about 99% to about 92% as more languages are added.

Interesting ideas

  • I1. Auxiliary data is added in order of measured similarity to the target and size-balanced, with an evaluation after each step; this exposed that the most distant language (English) actually hurt.
  • I2. Language identity is injected simply by putting a tag token at the start of the target sequence, so the model conditions on it without architectural changes.
  • I3. The way test material is elicited changes the result: translating from another language produced more natural variation than reading standard text. Benchmarks that echo the training norm therefore overestimate quality.
  • I4. The auxiliary task's own accuracy (LID recall) is reported next to the main metric, which shows when the helper signal starts to degrade as more languages are added.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When mixing extra treebanks or converted UD dialects into training, add them one at a time by measured closeness, cap each at the core set's size, and drop any that worsen held-out LAS.
  • I2 (tree): gparse could take a leading source/genre tag (treebank or register) so one parser can serve differently annotated or styled inputs while keeping the tag visible as an explanation.
  • I3 (answer): Build some test questions and fables by free retelling rather than by copying the training register, and report both scores so that brittleness to paraphrase becomes visible.
  • I4 (learn): Whenever an auxiliary label (genre, source parser, frame type) feeds training, track its accuracy separately so a decaying helper signal is caught before it silently harms parsing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GroningenGroningen · Netherlands70 works · 5,142 citations

Also on the byline: Fryske Akademy.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 4Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksMultilingual & cross-lingual NLP · 828 worksSpeech & spoken language · 480 worksLow-resource languages · 267 works

Cite

Cite as Reihaneh Amooie, Wietse de Vries, Yun Hao, Jelske Dijkstra, Matt Coler and Martijn Wieling. 2025. Evaluating Standard and Dialectal Frisian ASR: Multilingual Fine-tuning and Language Identification for Improved Low-resource Performance. arXiv:2502.04883.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.04883 · arXiv 2502.04883

@misc{amooie2025evaluating,
  title = {{Evaluating Standard and Dialectal Frisian ASR: Multilingual Fine-tuning and Language Identification for Improved Low-resource Performance}},
  author = {Reihaneh Amooie, Wietse {de Vries}, Yun Hao, Jelske Dijkstra, Matt Coler and Martijn Wieling},
  year = {2025},
  doi = {10.48550/arxiv.2502.04883},
  eprint = {2502.04883},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.04883}
}

References

17 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. William Chen, Brian Yan, Jiatong Shi, Yifan Peng, Soumi Maiti, and Shinji Watanabe, “Improving massively multilingual asr with auxiliary ctc objectives,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5. find
  2. Qianying Liu, Zhuo Gong, Zhengdong Yang, Yuhang Yang, Sheng Li, Chenchen Ding, Nobuaki Minematsu, Hao Huang, Fei Cheng, Chenhui Chu, et al., “Hierarchical softmax for end-to-end low-resource multilingual speech recognition,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5. find
  3. Hemant Yadav and Sunayana Sitaram, “A survey of multilingual models for automatic speech recognition,” arXiv preprint arXiv:2202.12576, 2022. ↗
  4. Yosuke Kashiwagi, Hayato Futami, Emiru Tsunoo, Siddhant Arora, and Shinji Watanabe, “Rapid language adaptation for multilingual e2e speech recognition using encoder prompting,” arXiv preprint arXiv:2406.12611, 2024. ↗
  5. Yoohwan Kwon and Soo-Whan Chung, “Mole: Mixture of language experts for multi-lingual automatic speech recognition,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5. find
  6. Austin Waters, Neeraj Gaur, Parisa Haghani, Pedro Moreno, and Zhongdi Qu, “Leveraging language id in multilingual end-to-end speech recognition,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 928–935. find
  7. Chao Zhang, Bo Li, Tara Sainath, Trevor Strohman, Sepand Mavandadi, Shuo-Yiin Chang, and Parisa Haghani, “Streaming end-to-end multilingual speech recognition with joint language identification,” arXiv preprint arXiv:2209.06058, 2022. ↗
  8. Zhao You, Shulin Feng, Dan Su, and Dong Yu, “Speechmoe2: Mixture-of-experts model with improved routing,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7217–7221. ↗
  9. William Fedus, Barret Zoph, and Noam Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022. ↗
  10. Alex Graves and Alex Graves, “Connectionist temporal classification,” Supervised sequence labelling with recurrent neural networks, pp. 61– 93, 2012. ↗
  11. R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 2020, pp. 4211–4215. ↗
  12. Søren Wichmann, Eric W Holman, Dik Bakker, and Cecil H Brown, “Evaluating linguistic distance measures,” Physica A: Statistical Mechanics and its Applications, vol. 389, no. 17, pp. 3632–3639, 2010. ↗
  13. Wietse De Vries, Martijn Bartelds, Malvina Nissim, and Martijn Wieling, “Adapting monolingual models: Data can be scarce when language similarity is high,” arXiv preprint arXiv:2105.02855, 2021. ↗
  14. Martijn Wieling, Teja Rebernik, and Jidde Jacobi, “Spraaklab: a mobile laboratory for collecting speech production data,” in Proceedings of the 20th International Congress of Phonetic Sciences. Guarant International, 2023, pp. 2060–2064. find
  15. Wilbert Heeringa, “Dialect variation in and around frisia; classification and relationships,” Us Wurk, vol. 54, no. 3-4, pp. 125–167, 2005. find
  16. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick Von Platen, Yatharth Saraf, Juan Pino, et al., “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” arXiv preprint arXiv:2111.09296, 2021. ↗
  17. A Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/amooie-2025-evaluating-standard-dialectal-frisian-asr/