Elamur Research⌕
☰
home / papers / 2026 / From LLM Prompts to Acoustic Baselines: A Scalable Pipeline for Under-Resourced Disfluent Code-Mixed Speech

From LLM Prompts to Acoustic Baselines: A Scalable Pipeline for Under-Resourced Disfluent Code-Mixed Speech

Summary

ASR systems for South Asian languages are rarely trained on audio that reflects real conversation, where speakers switch between their native language and English mid-sentence and also hesitate, repeat words or restart. Collecting and transcribing such speech is expensive. The authors therefore build BEHE-CMDisfl, a fully synthetic corpus of Bengali–English and Hindi–English conversational speech. Structured ChatGPT prompts fix speaker roles, topic, degree of code-mixing and disfluency rate, and the resulting dialogues are voiced with Indic Parler TTS. The corpus is tested on about 1.3 hours of Bengali–English audio with two ASR approaches. A Kaldi GMM-HMM system reaches 37.74% WER after phonetic normalization reduces transliteration variation, and it keeps disfluency tokens in its output. openai/whisper-small fails in zero-shot mode, hallucinating and looping. After brief LoRA fine-tuning it decodes stably and reaches 21.37% WER. The work offers a cheap, language-adaptable recipe for bootstrapping ASR in micro-resource code-mixed settings, although results on real speech have not yet been reported.

Key points

  • Corpus and pipeline: BEHE-CMDisfl covers Bengali–English and Hindi–English, with Bengali or Hindi as the matrix language and English embedded.
    • Text comes from zero- and few-shot ChatGPT prompting; audio comes from Indic Parler TTS.
    • Prompts request filled pauses, repetitions and restarts in roughly 15–25% of utterances.
  • Domains: everyday chat, customer-service dialogs, classroom talk, healthcare/teleconsultation, and social-media-style sports and entertainment commentary.
  • Kaldi GMM-HMM baseline (~1.3 h Bengali–English): 37.74% WER after phonetic normalization, with disfluency markers retained as lexical units.
  • Whisper adaptation: zero-shot openai/whisper-small hallucinated heavily and failed to emit end-of-sequence tokens. LoRA fine-tuning for 1,000 steps brought WER to 21.37% and improved recognition of rapid language switching.
  • Limitations and future work:
    • Evaluation is synthetic only; a 3–5 hour real-speech test set is planned.
    • The Hindi–English subset has not yet been tested for ASR.
    • Planned extensions include Wav2Vec2 adaptation and Tamil and Marathi.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5k9xcfdyrtev

@inproceedings{mitra2026llm,
  title = {{From LLM Prompts to Acoustic Baselines: A Scalable Pipeline for Under-Resourced Disfluent Code-Mixed Speech}},
  author = {Mitra, Anuran and Chakravarty, Anirvan and Mondal, Tapabrata and Bandyopadhyay, Sivaji},
  booktitle = {Proceedings of the SIGUL 2026 Joint Workshop with ELE, EURALI, and DCLRL: Towards Inclusivity and Equality: Language Resources and Technologies for Under-Resourced and Endangered Languages},
  year = {2026},
  doi = {10.63317/5k9xcfdyrtev},
  url = {https://doi.org/10.63317/5k9xcfdyrtev}
}

References

41 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jordi Adell, Antonio Bonafonte, and David Escudero. 2006. Disfluent speech analysis and synthesis: A preliminary approach. In Proc. 3rd Int. Conf. on Speech Prosody, pages 1–4. find
  2. Sadia Alam, Md Farhan Ishmam, Navid Hasin Alvee, Md Shahnewaz Siddique, Md Azam Hossain, and Abu Raihan Mostofa Kamal. 2024. Bnsentmix: A diverse bengali-english code-mixed dataset for sentiment analysis. arXiv preprint arXiv:2408.08964. find
  3. Robin Amann, Zhaolin Li, Barbara Bruno, and Jan Niehues. 2024. Augmenting automatic speech recognition models with disfluency detection. pages 224–231. find
  4. Peter Auer. 2013. Code-switching in conversation: Language, interaction and identity. Routledge. find
  5. L. Besacier, E. Barnard, A. Karpov, and T. Schultz. 2014. Automatic speech recognition for under-resourced languages: A survey. Speech Communication, 56:85–100. find
  6. Vineet Bhat, Preethi Jyothi, and Pushpak Bhattacharyya. 2023. Adversarial training for low-resource disfluency correction. arXiv preprint arXiv:2306.06384. find
  7. Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In Proc. O-COCOSDA, pages 1–5. find
  8. M. A. Dar and J. Pushparaj. 2026. A wav2vec2 model-based automatic speech recognition system for low-resource kashmiri language. International Journal of Speech Technology, 29(1):2. find
  9. A. Dhasmana, A. Srivastava, and D. Chiang. 2026. Dialect matters: Cross-lingual asr transfer for low-resource indic language varieties. arXiv preprint arXiv:2601.04373. find
  10. Anuj Diwan, Rakesh Vaideeswaran, Sanket Shah, Ankita Singh, Srinivasa Raghavan, Shreya Khare, Vinit Unni, Saurabh Vyas, Akash Rajpuria, Chiranjeevi Yarra, et al. 2021. Multilingual and code-switching asr challenges for low resource indian languages. arXiv preprint arXiv:2104.00235. — [arXiv:2104.00235](https://arxiv.org/abs/2104.00235) find
  11. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. In Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), pages 5206–5210. find
  12. Medibeng (revision J Godfrey and E Holliman. 1997. Switchboard-1 release 2: Linguistic data consortium. Switchboard: a user’s manual. find
  13. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, et al. 2014. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567. find our card
  14. T. D. Harya. 2018. Sociolinguistics: Code switching and code mixing. Lentera: Jurnal Ilmiah Kependidikan, 11(1):87–98. find
  15. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3. find
  16. Eunhee Kim. 2006. Reasons and motivations for code-mixing and code-switching. Issues in EFL, 4(1):43–61. find
  17. E. Kullmann. 2016. Speech-to-text for swedish using kaldi. Master’s thesis, KTH Royal Institute of Technology. find
  18. Rohit Kundu, Preethi Jyothi, and Pushpak Bhattacharyya. 2022. Zero-shot disfluency detection for indian languages. In Proceedings of the 29th international conference on computational linguistics, pages 4442–4454. find
  19. Y. Lacombe, V. Srivastav, and S. Gandhi. 2024. Parler-tts. GitHub repository. find
  20. D. Lyth and S. King. 2024. Natural language guidance of high-fidelity text-to-speech with synthetic annotations. arXiv preprint arXiv:2402.01912. — [arXiv:2402.01912](https://arxiv.org/abs/2402.01912) find
  21. M. Mohri, F. Pereira, and M. Riley. 2008. Speech recognition with weighted finite-state transducers. In Springer Handbook of Speech Processing, pages 559–584. Springer. find
  22. A. Pandey, B. M. L. Srivastava, and S. Sitaram. 2018. Adapting monolingual resources for code-mixed hindi-english speech recognition. In Proc. IEEE ICASSP. find
  23. Braja Gopal Patra, Dipankar Das, and Amitava Das. 2018. Sentiment analysis of code-mixed indian languages: An overview of sail_code-mixed shared task@ icon-2017. arXiv preprint arXiv:1803.06745. — [arXiv:1803.06745](https://arxiv.org/abs/1803.06745) find
  24. Aryan Paul, Tapabrata Mondal, Dipankar Das, and Sivaji Bandyopadhyay. 2025. Generating and analyzing disfluency in a code-mixed setting. Recent Advances in Natural Language Processing (RANLP), pages 915–924. find
  25. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, et al. 2011. The kaldi speech recognition toolkit. In Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). find
  26. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR. find our card
  27. Amrit Romana, Minxue Niu, Matthew Perez, and Emily Mower Provost. 2024. Fluencybank timestamped: An updated data set for disfluency detection and automatic intended speech recognition. Journal of Speech, Language, and Hearing Research, 67(11):4203–4215. find
  28. Jivnesh Sandhan, Ayush Daksh, Om Adideva Paranjay, Laxmidhar Behera, and Pawan Goyal. 2022. Prabhupadavani: A code-mixed speech translation data for 25 languages. arXiv preprint arXiv:2201.11391. find
  29. Melissa G Moyer. 2002. Bilingual speech: A typology of code-mixing. find
  30. Ashwin Sankar, Yoach Lacombe, Sherry Thomas, Praveen Srinivasa Varadhan, Sanchit Gandhi, and Mitesh M Khapra. 2025. Rasmalai: Resources for adaptive speech modeling in indian languages with accents and intonations. arXiv preprint arXiv:2505.18609. — [arXiv:2505.18609](https://arxiv.org/abs/2505.18609) find
  31. Md Nishat Raihan, Dhiman Goswami, Antara Mahmud, Antonios Anastasopoulos, and Marcos Zampieri. 2023. Sentmix-3l: A bangla-english-hindi code-mixed dataset for sentiment analysis. arXiv e-prints, pages arXiv–2310. find
  32. S Saranya, B Bharathi, S Gomathy Dhanya, and Aishwarya Krishnakumar. 2025. Real-time continuous tamil dialect speech recognition and summarization. Circuits, Systems, and Signal Processing, 44(4):2855–2881. find
  33. Elizabeth Shriberg. 2001. To ‘errrr’is human: ecology and acoustics of speech disfluencies. Journal of the international phonetic association, 31(1):153–169. find
  34. Vinay Singh, Deepanshu Vijay, Syed Sarfaraz Akhtar, and Manish Shrivastava. 2018. Named entity recognition for hindi-english code-mixed social media text. In Proceedings of the seventh named entities workshop, pages 27–35. find
  35. S. Sitaram, K. R. Chandu, S. K. Rallabandi, and A. W. Black. 2019. A survey of code-switched speech and language processing. arXiv preprint arXiv:1904.00784. — [arXiv:1904.00784](https://arxiv.org/abs/1904.00784) find
  36. Bao Thai, Robert Jimerson, Dominic Arcoraci, Emily Prud’hommeaux, and Raymond Ptucha. 2019. Synthetic data augmentation for improving low-resource asr. In 2019 IEEE Western New York Image and Signal Processing Workshop (WNYISPW), pages 1–9. find
  37. S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, et al. 2018. Espnet: End-to-end speech processing toolkit. In Proc. Interspeech. find
  38. Peng Xie, Xingyuan Liu, Tsz Wai Chan, Yequan Bie, Yangqiu Song, Yang Wang, Hao Chen, and Kani Chen. 2025. Switchlingua: The first large-scale multilingual and multi-ethnic code-switching dataset. arXiv preprint arXiv:2506.00087. — [arXiv:2506.00087](https://arxiv.org/abs/2506.00087) find
  39. Y. H. Yeo, Y. Hu, S. Gopal, Y. Peng, H. Liu, and E. S. Chng. 2026. Improving code-switching speech recognition with tts data augmentation. arXiv preprint arXiv:2601.00935. — [arXiv:2601.00935](https://arxiv.org/abs/2601.00935) find
  40. Emre Yılmaz, M. Andringa, S. Kingma, J. Dijkstra, F. van der Kuip, H. van de Velde, et al. 2016. Longitudinal speaker clustering and identification for frisian-dutch code-switching. In Proc. Interspeech, pages 3668–3672. find
  41. Vicky Zayats, Mari Ostendorf, and Hannaneh Hajishirzi. 2016. Disfluency detection using a bidirectional lstm. arXiv preprint arXiv:1604.03209. — [arXiv:1604.03209](https://arxiv.org/abs/1604.03209) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/mitra-2026-llm-prompts-acoustic-baselines-scalable/