Elamur Research⌕
☰
home / papers / 2025 / Speech Recognition for Automatically Assessing Afrikaans and isiXhosa Preschool Oral Narratives

Speech Recognition for Automatically Assessing Afrikaans and isiXhosa Preschool Oral Narratives

Paper · 2025 · Christiaan Jacobs, Annelien Smith, Daleen Klop et al.

Summary

Early oral language skills predict later reading. In crowded South African preschools, though, assessing 4–5-year-olds by hand is impractical. This work builds the first component of an automated assessment pipeline: speech recognition for spontaneous stories told by Afrikaans and isiXhosa preschoolers. The children produced the stories by describing picture sequences under the MAIN protocol. About five hours of child speech per language were manually aligned in Praat. To mimic a realistic new-language scenario, only a random 5-minute labelled subset is used for training. Starting from Whisper (small and medium), the authors compare several known child-ASR strategies: LoRA fine-tuning, adding adult speech, voice conversion and semi-supervised labelling. The largest gain comes from adult speech that matches the story domain, particularly when voice conversion is applied to it. Semi-supervised learning helps for both languages. LoRA helps Afrikaans but hurts isiXhosa, which Whisper never saw in pretraining. The paper's value is empirical: it tests prior strategies on two non-English languages at very young ages, a setting that is rarely studied.

Key points

  • Data: 154 Afrikaans-speaking and 157 isiXhosa-speaking children aged 4–5, drawn from 55 classrooms. Each language yields roughly 5 hours of aligned speech, and only a 5-minute subset serves as labelled training data.
  • Whisper medium fine-tuned on the 5 minutes reaches 47.4% WER on Afrikaans and 80.4% on isiXhosa. Without fine-tuning, WER is 88.1% and 105.6%. With the full training set (topline), it is 18.3% and 50.9%.
  • CER for the 5-minute models is 27.9% (Afrikaans) and 34.0% (isiXhosa). The gap is much smaller than for WER, so word-level scoring is especially harsh on agglutinative isiXhosa.
  • LoRA (rank 32 on the attention projections) improves Afrikaans from 47.4% to 41.1% but degrades isiXhosa from 80.4% to 88.7%.
  • isiXhosa is absent from Whisper's pretraining data, so the Swahili language token is used in its place. Decoding uses beam size 10 and a length penalty of −0.5 to curb repetition loops triggered by stuttering and false starts.

Interesting ideas

  • I1. Reporting character-level error alongside word-level error shows whether a language is failing badly or is just being scored harshly because its words are long and morphologically rich.
  • I2. Every result is placed between two anchors: an unadapted lower bound and a full-data topline. This makes it clear how much of the achievable gap each strategy actually closes.
  • I3. The most useful extra data was content that matched the story domain, even though it came from adults rather than children. Topical match mattered more than speaker match.
  • I4. For a language the base model never saw, borrowing the tag of a distant but available language gave fine-tuning a workable starting point.
  • I5. A mild length penalty at decoding time stops disfluent input from triggering runaway repeated output.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Report lemma- and morpheme-level accuracy alongside whole-token accuracy, so that errors on long derived or compound words are not mistaken for general tagger failure.
  • I2 (learn): Evaluate each repair rule or silver-data batch as the fraction of the gap closed between the untrained parser and a gold-trained one, not as a raw score.
  • I3 (answer): When building training material for the fable and tale readers, prefer texts that retell the same kinds of stories over generic text with merely similar style.
  • I4 (tree): When parsing a genre or UD dialect with no dedicated treebank, start gparse from the closest existing treebank's configuration and then adapt, rather than training from scratch.
  • I5 (back): Add a similar penalty to the graph → text generator, so that noisy or looping meaning graphs do not produce repeated phrases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🎓Stellenbosch UniversityStellenbosch · South Africa17 works · 24,141 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSignal Processing · 452 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 480 works

Cite

Cite as Christiaan Jacobs, Annelien Smith, Daleen Klop, Ondřej Klejch, Febe de Wet and Herman Kamper. 2025. Speech Recognition for Automatically Assessing Afrikaans and isiXhosa Preschool Oral Narratives. arXiv:2501.06478.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.06478 · arXiv 2501.06478

@misc{jacobs2025speech,
  title = {{Speech Recognition for Automatically Assessing Afrikaans and isiXhosa Preschool Oral Narratives}},
  author = {Christiaan Jacobs, Annelien Smith, Daleen Klop, Ondvrej Klejch, Febe {de Wet} and Herman Kamper},
  year = {2025},
  doi = {10.48550/arxiv.2501.06478},
  eprint = {2501.06478},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.06478}
}

References

46 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. van Staden, K. Roux, and M. Tshele, “PIRLS 2021: South African children’s literacy achievement,” Centre for Evaluation and Assessment, 2023. find
  2. Y. D. Chiu, “The simple view of reading across development: Prediction of Grade 3 reading comprehension from prekindergarten skills,” Remedial and Special Education, 2018. ↗
  3. H. N. Hjetland, E. I. Brinchmann, R. Scherer, C. Hulme, and M. Melby-Lervåg, “Preschool pathways to reading comprehension: A systematic meta-analytic review,” Educational Research Review, 2020. ↗
  4. S. Babayiğit, S. Roulstone, and Y. Wren, “Linguistic comprehension and narrative skills predict reading ability: A 9-year longitudinal study,” British Journal of Educational Psychology, 2021. ↗
  5. A. Schick and G. Melzi, “The development of children’s oral narratives across contexts,” Early Education, 2010. find
  6. E. Reese, D. Leyva, A. Sparks, and W. Grolnick, “Maternal elaborative reminiscing increases low-income children’s narrative skills relative to dialogic reading,” Early Education and Development, 2010. find
  7. J. V. Oakhill and K. Cain, “The precursors of reading ability in young readers: Evidence from a four-year longitudinal study,” Scientific Studies of Reading, 2012. ↗
  8. N. Gardner-Neblett and I. U. Iruka, “Oral narrative skills: Explaining the language-emergent literacy link by race/ethnicity and SES,” Developmental Psychology, 2015. ↗
  9. A. P. Kaiser and M. Y. Roberts, “Advances in early communication and language intervention,” Journal of Early Intervention, 2011. ↗
  10. K. L. Carson, G. T. Gillon, and T. M. Boustead, “Classroom phonological awareness instruction and literacy outcomes in the first year of school,” Language, Speech, and Hearing Services in Schools, 2013. ↗
  11. G. Gillon, B. McNeill, A. Scott, A. Denston, L. Wilson, K. Carson, and A. H. Macfarlane, “A better start to literacy learning: Findings from a teacher-implemented intervention in children’s first year at school,” Reading and Writing, 2019. find
  12. S. Lee, A. Potamianos, and S. Narayanan, “Acoustics of children’s speech: Developmental changes of temporal and spectral parameters,” Acoustical Society of America, 1999. find
  13. M. Gerosa, D. Giuliani, and F. Brugnara, “Acoustic variability and automatic recognition of children’s speech,” Speech Communication, 2007. find
  14. H. Veeramani, N. B. Shankar, A. Johnson, and A. Alwan, “Towards automatically assessing children’s picture description tasks,” in Proc. SLaTE, 2023. find
  15. A. Johnson, H. Veeramani, N. Balaji Shankar, and A. Alwan, “An equitable framework for automatically assessing children’s oral narrative language abilities,” in Proc. Interspeech, 2023. find
  16. G. Yeung and A. Alwan, “On the difficulties of automatic speech recognition for kindergarten-aged children,” in Proc. Interspeech, 2018. find
  17. F. Claus, H. G. Rosales, R. Petrick, H.-U. Hain, and R. Hoffmann, “A survey about databases of children’s speech,” in Proc. Interspeech, 2013. find
  18. Y. Zhang, Z. Yue, T. Patel, and O. Scharenborg, “Improving child speech recognition with augmented child-like speech,” in Proc. Interspeech, 2024. find
  19. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proc. ICML., 2023. ↗
  20. R. Fan, N. B. Shankar, and A. Alwan, “Benchmarking children’s ASR with supervised and self-supervised speech foundation models,” in Proc. Interspeech, 2024. find
  21. Z. Song, J. Zhuo, Y. Yang, Z. Ma, S. Zhang, and X. Chen, “LoRA-Whisper: Parameter-efficient and extensible multilingual ASR,” in Proc. Interspeech, 2024. find
  22. P. G. Shivakumar and P. Georgiou, “Transfer learning from adult to children for speech recognition: Evaluation, analysis and recommendations,” Computer Speech and Language, 2020. ↗
  23. S. Shahnawazuddin, N. Adiga, K. Kumar, A. Poddar, and W. Ahmad, “Voice conversion based data augmentation to improve children’s speech recognition in limited data scenario,” in Proc. Interspeech, 2020. find
  24. L. Lamel, J.-L. Gauvain, and G. Adda, “Lightly supervised and unsupervised acoustic model training,” Computer Speech and Language, 2002. ↗
  25. N. Gagarina, D. Klop, S. Kunnari, K. Tantele, T. Välimaa, U. Bohnacker, and J. Walters, “MAIN: Multilingual assessment instrument for narratives – revised version,” ZAS Papers in Linguistics, 2019. ↗
  26. K. Cain, S. O’Carroll, J. Oakhill, D. Klop, M. Visser, A. Smith, and A. Swart, “Exploring the impact of a story-based teacher training programme on language and early literacy in 4- and 5-year-olds,” Wordworks, Cape Town, 2024. ↗
  27. P. Boersma and D. Weenink, “Praat: Doing phonetics by computer,” http://www.praat.org/, 2022, version 6.2.09. ↗
  28. W. Liu, Y. Qin, Z. Peng, and T. Lee, “Sparsely shared LoRA on Whisper for child speech recognition,” in Proc. ICASSP, 2024. find
  29. R. Jain, A. Barcovschi, M. Y. Yiwere, P. Corcoran, and H. Cucu, “Exploring native and non-native English child speech recognition with Whisper,” IEEE Access, 2024. ↗
  30. A. Babu et al., “XLS-R: Self-supervised cross-lingual speech representation learning at scale,” in Proc. Interspeech, 2022.
  31. E. Barnard, M. Davel, C. van Heerden, F. Wet, and J. Badenhorst, “The NCHLT speech corpus of the South African languages,” in Proc. SLTU, 2014. ↗
  32. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in Proc. ICLR, 2022.
  33. D. Elenius and M. Blomberg, “Adaptation and normalization experiments in speech recognition for 4 to 8 year old children,” in Proc. Interspeech, 2005. find
  34. J. Fainberg, P. Bell, M. Lincoln, and S. Renals, “Improving children’s speech recognition through out-of-domain data augmentation,” in Proc. Interspeech, 2016. find
  35. P. Bell, J. Fainberg, O. Klejch, J. Li, S. Renals, and P. Swietojanski, “Adaptation algorithms for neural network-based speech recognition: An overview,” IEEE Open Journal of Signal Processing, 2020. ↗
  36. D. K. Singh, P. P. Amin, H. B. Sailor, and H. A. Patil, “Data augmentation using CycleGAN for end-to-end children ASR,” in Proc. EUSIPCO, 2021. find
  37. Z. Shuyang, M. Singh, A. Woubie, and R. Karhila, “Data augmentation for children ASR and child-adult speaker classification using voice conversion methods,” in Proc. Interspeech, 2023. find
  38. M. Baas and H. Kamper, “Voice conversion for stuttered speech, instruments, unseen languages and textually described voices,” Communications in Computer and Information Science, 2023. find
  39. M. Baas, B. van Niekerk, and H. Kamper, “Voice conversion with just nearest neighbors,” in Proc. Interspeech, 2023. find
  40. M. Baas and H. Kamper, “Voice conversion can improve ASR in very low-resource settings,” in Proc. Interspeech, 2022. find
  41. S. Thomas, M. L. Seltzer, K. Church, and H. Hermansky, “Deep neural network features and semi-supervised training for low resource speech recognition,” in Proc. ICASSP, 2013. find
  42. F. Weninger, F. Mana, R. Gemello, J. Andrés-Ferrer, and P. Zhan, “Semi-supervised learning with data augmentation for end-to-end ASR,” in Proc. Interspeech, 2020. find
  43. E. Wallington, B. Kershenbaum, O. Klejch, and P. Bell, “On the learning dynamics of semi-supervised training for ASR,” in Proc. Interspeech, 2021. find
  44. J. Wang, Y. Zhu, R. Fan, W. Chu, and A. Alwan, “Low resource German ASR with untranscribed data spoken by non-native children – Interspeech 2021 shared task SPAPL system,” in Proc. Interspeech, 2021. ↗
  45. Y. Tao, S. G. Mitsven, L. K. Perry, D. S. Messinger, and M.-L. Shyu, “Audio-based group detection for classroom dynamics analysis,” in Proc. ICDMW, 2019. find
  46. S. L. Pugh, S. K. Subburaj, A. R. Rao, A. E. B. Stewart, J. Andrews-Todd, and S. K. D’Mello, “Say what? Automatic modeling of collaborative problem solving skills from student speech in the wild,” in Proc. EDM, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jacobs-2025-speech-recognition-automatically-assessing-afrikaans/