Elamur Research⌕
☰
home / papers / 2026 / Bengali-English and Hindi-English Code Mixed Speech Data with Disfluencies

Bengali-English and Hindi-English Code Mixed Speech Data with Disfluencies

Paper · 2026 · Anuran Mitra, Tapabrata Mondal, Anirvan Chakravarty et al.

Summary

Most Bengali–English and Hindi–English speech corpora contain clean, scripted speech. Real Indian conversation mixes languages and includes hesitations, so ASR systems built on fluent data handle it poorly. Recording such natural speech is expensive, particularly when only about an hour of data is realistic. The authors created BEHE-CMDisfl, a synthetic corpus built in two stages. First, ChatGPT (GPT-5) was prompted, zero-shot and few-shot, to write informal multi-speaker dialogues covering everyday, service, classroom, healthcare and social-media topics. Bengali or Hindi served as the matrix language and English as the embedded one, and the prompts asked for injected filled pauses, repetitions and restarts. Second, Indic Parler TTS converted the dialogues to audio. To check whether the data is usable, they trained Kaldi GMM-HMM models on a 1.3-hour Bengali–English subset. The best word error rate was 37.74%, and most of the gains came from consistent lexicon entries and normalized transliteration rather than from more complex models. The recognized output kept the disfluency tokens instead of discarding them. The resource offers a controlled testbed for disfluency-aware code-mixed ASR, but results still need checking on real speech.

Key points

  • Pipeline: LLM-generated code-mixed disfluent dialogues (ChatGPT, GPT-5) were voiced with AI4Bharat's Indic Parler TTS. The corpus covers Bengali–English (BE) and Hindi–English (HE). It extends the text-only MixFluent work into audio.
  • Baselines: Kaldi Monophone, Triphone and LDA-MLLT models were trained on a 1.3 h BE subset. The Triphone model reached 39.23% WER. Widening the decoding beam gave the best result, 37.74% WER. HE ASR experiments are still planned.
  • Normalization matters: Unifying inconsistent romanizations (e.g., "ar" vs "aar") and keeping the lexicon consistent mattered more than model complexity.
  • Disfluencies are recognized: When they had lexicon entries, hesitation sounds like "um", self-corrections like "i mean" and doubled words like "oh oh" showed up correctly in the decoded transcripts.
  • Limitations: The audio is synthetic, and no significance testing was done. The authors cite a 47.10% Wav2Vec2 WER from a different study using 10 h of data, but this is not a controlled comparison. Future plans include a 3–5 h real-speech gold set to measure the sim-to-real gap and extension to Tamil, Marathi and Telugu.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Jadavpur UniversityKolkata · India8 works · 336 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,379 worksMultilingual & cross-lingual NLP · 882 worksSpeech & spoken language · 519 works

Cite

Cite as Anuran Mitra, Tapabrata Mondal, Anirvan Chakravarty and Sivaji Bandyopadhyay. 2026. Bengali-English and Hindi-English Code Mixed Speech Data with Disfluencies. Proceedings of the 8th Workshop on Indian Language Data: Resources and Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/254iufgehzzc

@inproceedings{mitra2026bengali,
  title = {{Bengali-English and Hindi-English Code Mixed Speech Data with Disfluencies}},
  author = {Anuran Mitra, Tapabrata Mondal, Anirvan Chakravarty and Sivaji Bandyopadhyay},
  booktitle = {Proceedings of the 8th Workshop on Indian Language Data: Resources and Evaluation},
  year = {2026},
  doi = {10.63317/254iufgehzzc},
  url = {https://doi.org/10.63317/254iufgehzzc}
}

References

38 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jordi Adell, Antonio Bonafonte, and David Escudero. 2006. Disfluent speech analysis and synthesis: A preliminary approach. In Proc. 3rd Int. Conf. on Speech Prosody, pages 1–4. \#\# 7. Summary 7.1. find
  2. Summary of Contributions We present BEHE-CMDisfl, a synthetic, reproducible Bengali–English and Hindi–English code-mixed speech corpus with explicit annotations for disfluency phenomena and language segments. The corpus fills an important gap for multilingual, disfluent speech modeling in Indic cont… find
  3. Robin Amann, Zhaolin Li, Barbara Bruno, and Jan Niehues. 2024. Augmenting automatic speech recognition models with disfluency detection. pages 224–231. find
  4. Rohit Kundu, Preethi Jyothi, and Pushpak Bhattacharyya. 2022. Zero-shot disfluency detection for indian languages. In Proceedings of the 29th international conference on computational linguistics, pages 4442–4454. find
  5. Peter Auer. 2013. Code-switching in conversation: Language, interaction and identity. Routledge. find
  6. Y. Lacombe, V. Srivastav, and S. Gandhi. 2024. Parler-tts. GitHub repository. find
  7. L. Besacier, E. Barnard, A. Karpov, and T. Schultz. 2014. Automatic speech recognition for under-resourced languages: A survey. Speech Communication, 56:85–100. find
  8. D. Lyth and S. King. 2024. Natural language guidance of high-fidelity text-to-speech with synthetic annotations. arXiv preprint arXiv:2402.01912. — [arXiv:2402.01912](https://arxiv.org/abs/2402.01912) find
  9. Vineet Bhat, Preethi Jyothi, and Pushpak Bhattacharyya. 2023. Adversarial training for low-resource disfluency correction. arXiv preprint arXiv:2306.06384. find
  10. M. Mohri, F. Pereira, and M. Riley. 2008. Speech recognition with weighted finite-state transducers. In Springer Handbook of Speech Processing, pages 559–584. Springer. find
  11. Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In Proc. O-COCOSDA, pages 1–5. find
  12. Melissa G Moyer. 2002. Bilingual speech: A typology of code-mixing. find
  13. Md Nishat Raihan, Dhiman Goswami, Antara Mahmud, Antonios Anastasopoulos, and Marcos Zampieri. 2023. Sentmix-3l: A bangla-english-hindi code-mixed dataset for sentiment analysis. arXiv e-prints, pages arXiv–2310. find
  14. M. A. Dar and J. Pushparaj. 2026. A wav2vec2 model-based automatic speech recognition system for low-resource kashmiri language. International Journal of Speech Technology, 29(1):2. find
  15. A. Dhasmana, A. Srivastava, and D. Chiang. 2026. Dialect matters: Cross-lingual asr transfer for low-resource indic language varieties. arXiv preprint arXiv:2601.04373. find
  16. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. In Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), pages 5206–5210. find
  17. Anuj Diwan, Rakesh Vaideeswaran, Sanket Shah, Ankita Singh, Srinivasa Raghavan, Shreya Khare, Vinit Unni, Saurabh Vyas, Akash Rajpuria, Chiranjeevi Yarra, et al. 2021. Multilingual and code-switching asr challenges for low resource indian languages. arXiv preprint arXiv:2104.00235. — [arXiv:2104.00235](https://arxiv.org/abs/2104.00235) find
  18. A. Pandey, B. M. L. Srivastava, and S. Sitaram. 2018. Adapting monolingual resources for code-mixed hindi-english speech recognition. In Proc. IEEE ICASSP. find
  19. Braja Gopal Patra, Dipankar Das, and Amitava Das. 2018. Sentiment analysis of code-mixed indian languages: An overview of sail_code-mixed shared task@ icon-2017. arXiv preprint arXiv:1803.06745. — [arXiv:1803.06745](https://arxiv.org/abs/1803.06745) find
  20. Medibeng (revision J Godfrey and E Holliman. 1997. Switchboard-1 release 2: Linguistic data consortium. Switchboard: a user’s manual. find
  21. Aryan Paul, Tapabrata Mondal, Dipankar Das, and Sivaji Bandyopadhyay. 2025. Generating and analyzing disfluency in a code-mixed setting. Recent Advances in Natural Language Processing (RANLP), pages 915–924. find
  22. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, et al. 2014. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567.
  23. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, et al. 2011. The kaldi speech recognition toolkit. In Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). find
  24. T. D. Harya. 2018. Sociolinguistics: Code switching and code mixing. Lentera: Jurnal Ilmiah Kependidikan, 11(1):87–98. find
  25. Eunhee Kim. 2006. Reasons and motivations for code-mixing and code-switching. Issues in EFL, 4(1):43–61. find
  26. Amrit Romana, Minxue Niu, Matthew Perez, and Emily Mower Provost. 2024. Fluencybank timestamped: An updated data set for disfluency detection and automatic intended speech recognition. Journal of Speech, Language, and Hearing Research, 67(11):4203–4215. find
  27. E. Kullmann. 2016. Speech-to-text for swedish using kaldi. Master’s thesis, KTH Royal Institute of Technology. find
  28. Jivnesh Sandhan, Ayush Daksh, Om Adideva Paranjay, Laxmidhar Behera, and Pawan Goyal. 2022. Prabhupadavani: A code-mixed speech translation data for 25 languages. arXiv preprint arXiv:2201.11391. find
  29. S Saranya, B Bharathi, S Gomathy Dhanya, and Aishwarya Krishnakumar. 2025. Real-time continuous tamil dialect speech recognition and summarization. Circuits, Systems, and Signal Processing, 44(4):2855–2881. find
  30. Elizabeth Shriberg. 2001. To ‘errrr’is human: ecology and acoustics of speech disfluencies. Journal of the international phonetic association, 31(1):153–169. find
  31. Vinay Singh, Deepanshu Vijay, Syed Sarfaraz Akhtar, and Manish Shrivastava. 2018. Named entity recognition for hindi-english code-mixed social media text. In Proceedings of the seventh named entities workshop, pages 27–35. find
  32. S. Sitaram, K. R. Chandu, S. K. Rallabandi, and A. W. Black. 2019. A survey of code-switched speech and language processing. arXiv preprint arXiv:1904.00784. — [arXiv:1904.00784](https://arxiv.org/abs/1904.00784) find
  33. Bao Thai, Robert Jimerson, Dominic Arcoraci, Emily Prud’hommeaux, and Raymond Ptucha. 2019. Synthetic data augmentation for improving low-resource asr. In 2019 IEEE Western New York Image and Signal Processing Workshop (WNYISPW), pages 1–9. find
  34. S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, et al. 2018. Espnet: End-to-end speech processing toolkit. In Proc. Interspeech. find
  35. Peng Xie, Xingyuan Liu, Tsz Wai Chan, Yequan Bie, Yangqiu Song, Yang Wang, Hao Chen, and Kani Chen. 2025. Switchlingua: The first large-scale multilingual and multi-ethnic code-switching dataset. arXiv preprint arXiv:2506.00087. — [arXiv:2506.00087](https://arxiv.org/abs/2506.00087) find
  36. Y. H. Yeo, Y. Hu, S. Gopal, Y. Peng, H. Liu, and E. S. Chng. 2026. Improving code-switching speech recognition with tts data augmentation. arXiv preprint arXiv:2601.00935. — [arXiv:2601.00935](https://arxiv.org/abs/2601.00935) find
  37. Emre Yılmaz, M. Andringa, S. Kingma, J. Dijkstra, F. van der Kuip, H. van de Velde, et al. 2016. Longitudinal speaker clustering and identification for frisian-dutch code-switching. In Proc. Interspeech, pages 3668–3672. find
  38. Vicky Zayats, Mari Ostendorf, and Hannaneh Hajishirzi. 2016. Disfluency detection using a bidirectional lstm. arXiv preprint arXiv:1604.03209. — [arXiv:1604.03209](https://arxiv.org/abs/1604.03209) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mitra-2026-bengali-english-hindi-english-code/