Elamur Research⌕
☰
home / papers / 2025 / SAGE: Spliced-Audio Generated Data for Enhancing Foundational Models in Low-Resource Arabic-English Code-Switched Speech Recognition

SAGE: Spliced-Audio Generated Data for Enhancing Foundational Models in Low-Resource Arabic-English Code-Switched Speech Recognition

Summary

The paper tackles automatic transcription of Arabic dialects (DA) and of speech that mixes Arabic with English (CS). Real CS recordings are scarce, and most Arabic resources cover only Modern Standard Arabic. The authors first compare base-size SSL encoders fine-tuned on about 1,217 hours of monolingual Arabic and English audio. mHuBERT-147 performs best, beating the larger XLSR. They then propose SAGE, which builds artificial CS audio from monolingual recordings. Word boundaries are found with monolingual ASR and Viterbi alignment. A short run of words from one language is cut out and inserted into an utterance in the other language, in both directions. The already fine-tuned model is trained again on this material, mixed with original data in an Experience Replay-style scheme to limit forgetting. An out-of-domain 3-gram LM is then added, followed by few-shot tuning on one hour each of ZAEBUC and ArzEn. The final ~97M-parameter model beats USM, Whisper-large-v2 and MMS on CS benchmarks. This shows that targeted synthetic data can matter more than model size.

Key points

  • Training data: 449.2 h of English (AMI, English CommonVoice, TED-LIUM2) and 768.1 h of Arabic (Arabic CommonVoice plus 666.1 h of proprietary Saudi, Egyptian and Gulf conversational speech). Evaluation uses the ZAEBUC-spoken and ArzEn CS corpora.
  • In zero-shot comparison, mHuBERT-147 (97.3M parameters) reaches 33.9% overall mean WER. XLSR (318.7M) gets 39.1% and Whisper-base gets 52.2%.
  • SAGE inserts fragments of 2–4 words at a word gap or at an utterance edge. The result is volume-normalised and smoothed at the joins. SAGE fine-tuning cuts mean CS WER by 7.8 points absolute.
  • Training on SAGE alone strongly lowers WER on English-only and CS utterances but slightly raises it on Arabic-only ones. Mixing in monolingual data recovers Arabic performance.
  • The 3-gram KenLM lowers overall mean WER from 31.7% to 26.6%.
  • Few-shot tuning on 1 h each of ZAEBUC and ArzEn brings CS mean WER from 36.0% to 31.1%. This beats USM (36.6%), Whisper large-v2 (39.5%) and MMS (65.6%). Using the full in-domain training sets instead gave a worse result (32.3%).

Interesting ideas

  • I1. Mixed-condition training data can be built by cutting short spans from plentiful single-condition sources at aligned boundaries and inserting them into other sources, so the amount of synthetic data is not capped by scarce real examples.
  • I2. The synthetic set is mixed with replayed original data in an Experience Replay style, which keeps gains on the new condition without losing the old one.
  • I3. Errors are broken down by utterance type (Arabic-only, English-only, mixed), which exposes trade-offs that averages hide; tiny subsets with near-meaningless scores are explicitly set aside.
  • I4. An ablation checks whether gains come simply from more data: a small balanced in-domain sample beat using all in-domain data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): rare English constructions (parentheticals, fronted clauses, embedded quotes) could be created by grafting subtrees between UD treebank sentences at clause boundaries, yielding extra silver training trees for the parsers.
  • I2 (learn): when the perceptron parser or gparse is retrained on silver or new-domain data, a fixed share of gold treebank sentences could be replayed in each batch to guard against regression.
  • I3 (learn): evaluation could report scores per phenomenon slice and flag slices with too few items instead of folding them into a single LAS.
  • I4 (learn): every silver-data gain could be paired with a control that adds the same amount of unfiltered data, confirming that the selection matters rather than the volume.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Speech & spoken language Low-resource languages Semitic languages

 Institutions

Also on the byline: Emotech.

Cite

Cite as Muhammad Umar Farooq and Oscar Saz. 2025. SAGE: Spliced-Audio Generated Data for Enhancing Foundational Models in Low-Resource Arabic-English Code-Switched Speech Recognition. arXiv:2506.22143.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2506.22143 · arXiv 2506.22143

@misc{farooq2025sage,
  title = {{SAGE: Spliced-Audio Generated Data for Enhancing Foundational Models in Low-Resource Arabic-English Code-Switched Speech Recognition}},
  author = {Muhammad Umar Farooq and Oscar Saz},
  year = {2025},
  doi = {10.48550/arxiv.2506.22143},
  eprint = {2506.22143},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.22143}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Thien Nguyen et al., “Optimizing bilingual neural transducer with synthetic code-switching text generation,” 2022. find
  2. Mumtaz Begum Mustafa et al., “Code-switching in automatic speech recognition: The issues and future directions,” Applied Sciences, vol. 12, no. 19, 2022. find
  3. Alec Radford et al., “Robust speech recognition via large-scale weak supervision,” 2022. our card
  4. Vineel Pratap et al., “Scaling speech technology to 1,000+ languages,” 2023. our card
  5. Lingxuan Ye et al., “Improving recognition of out-of-vocabulary words in e2e code-switching asr by fusing speech generation methods,” in Interspeech 2022, 2022, pp. 3163– 3167. find
  6. Chenpeng Du et al., “Data augmentation for end-to-end code-switching speech recognition,” in IEEE SLT, 2021, pp. 194– 200. find
  7. Muhammad Umar Farooq et al., “Enhancing large vocabulary continuous speech recognition system for urdu-english conversational code-switched speech,” in IEEE Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O-COCOSDA), 2020, pp. 155–159. find
  8. Tolulope Ogunremi et al., “Multilingual self-supervised speech representations improve the speech recognition of low-resource African languages with codeswitching,” in Proceedings of the 6th Workshop on Computational Approaches to Linguistic Code-Switching, Singapore, Dec. 2023, pp. 83–88, Association for Computational Linguistics. find
  9. Enes Yavuz Ugan et al., “DECM: Evaluating bilingual ASR performance on a code-switching/mixing benchmark,” in Proceedings of LREC-COLING 2024, Torino, Italia, May 2024, pp. 4468–4475, ELRA and ICCL. find
  10. Dau-Cheng Lyu et al., “Seame: a mandarin-english code-switching speech corpus in south-east asia,” in Interspeech 2010, 2010, pp. 1986–1989. find
  11. Joyce Y. C. Chan et al., “Development of a cantonese-english code-mixing speech corpus,” in Interspeech 2005, 2005, pp. 1533–1536. find
  12. Anuj Diwan et al., “Mucs 2021: Multilingual and code-switching asr challenges for low resource indian languages,” in Interspeech 2021, 2021, pp. 2446–2450. find
  13. Zheng Liang et al., “Improving code-switching and name entity recognition in asr with speech editing based data augmentation,” in Interspeech 2023, 2023, pp. 919–923. find
  14. Alexis Conneau et al., “Unsupervised cross-lingual representation learning for speech recognition,” 2020. find
  15. Rehan Ahmad et al., “Progressive unsupervised domain adaptation for asr using ensemble models and multi-stage training,” in ICASSP, 2024, pp. 11466–11470. [27] Hsuan Su et al., “Task arithmetic can mitigate synthetic-toreal gap in automatic speech recognition,” in Proceedings of the EMNLP, Miami, Florida, USA, Nov. 2024, pp. 8905–8915, Association for Computational Linguistics. [28] Amir Hussein et al., “Speech collage: code-switched audio generation by collaging monolingual corpora,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12006–12010. [29] Ankita Pasad et al., “Comparative layer-wise analysis of self-supervised speech models,” in ICASSP, 2023, pp. 1–5. [30] Steve Renals et al., “Recognition and understanding of meetings the ami and amida projects,” in IEEE ASRU, 2007, pp. 238–247. [31] R. Ardila et al., “Common voice: A massively-multilingual speech corpus,” in Proceedings of the LREC, 2020, pp. 4211– 4215. our card
  16. Han Zhu et al., “Wav2vec-s: Semi-supervised pre-training for low-resource asr,” in Interspeech 2022, 2022, pp. 4870–4874. [32] Anthony Rousseau et al., “TED-LIUM: an automatic speech recognition dedicated corpus,” in Proceedings of the LREC, 2012, pp. 125–129. find
  17. Hyungchan Song et al., “Exploring wavlm on speech enhancement,” in 2022 IEEE Spoken Language Technology Workshop (SLT), 2023, pp. 451–457. [33] Kenneth Heafield et al., “Scalable modified Kneser-Ney language model estimation,” in Proceedings of the ACL, Aug. 2013, pp. 690–696. our card
  18. Biao Fu et al., “wav2vec-S: Adapting pre-trained speech models for streaming,” in Findings of the ACL 2024, Bangkok, Thailand, Aug. 2024, Association for Computational Linguistics. [34] Wei-Ning Hsu et al., “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 29, pp. 3451–3460, Oct. 2021. our card
  19. Kuan-Po Huang et al., “Zero resource code-switched speech benchmark using speech utterance pairs for multiple spoken languages,” in ICASSP 2024, 2024, pp. 10006–10010. [35] Alexei Baevski et al., “wav2vec 2.0: A framework for self-supervised learning of speech representations,” 2020. our card
  20. Injy Hamed et al., “ZAEBUC-spoken: A multilingual multidialectal Arabic-English speech corpus,” in Proceedings of the LREC-COLING 2024. 2024, pp. 17770–17782, ELRA and ICCL. find
  21. Injy Hamed et al., “ArzEn: A speech corpus for code-switched Egyptian Arabic-English,” in Proceedings of the LREC, Marseille, France, May 2020, pp. 4237–4246, European Language Resources Association. find
  22. Rosana Ardila et al., “Common voice: A massively-multilingual speech corpus,” in Proceedings of the Twelfth Language Resources and Evaluation Conference, Marseille, France, May 2020, pp. 4218–4222, European Language Resources Association. our card
  23. Ahmed Ali et al., “The mgb-2 challenge: Arabic multi-dialect broadcast media recognition,” 2019. our card
  24. Hamdy Mubarak et al., “QASR: QCRI aljazeera speech resource a large scale annotated Arabic speech corpus,” in ACL, Online, Aug. 2021, pp. 2274–2285, Association for Computational Linguistics. find
  25. David Rolnick et al., “Experience replay for continual learning,” in NeurIPS. 2019, vol. 32, Curran Associates, Inc. [36] Marcely Zanon Boito et al., “mhubert-147: A compact multilingual hubert model,” in Interspeech 2024, 2024, pp. 3939–3943. [37] Bashar Talafha et al., “N-shot benchmarking of whisper on diverse arabic speech recognition,” in Interspeech 2023, 2023, pp. 5092–5096. [38] Micro Ravanelli et al., “SpeechBrain: A general-purpose speech toolkit,” 2021, arXiv:2106.04624. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/farooq-2025-sage-spliced-audio-generated-data/