Elamur Research⌕
☰
home / papers / 2026 / A Benchmark Dataset and Comparative Evaluation of Phonemized and Romanized Urdu for Text-to-Speech

A Benchmark Dataset and Comparative Evaluation of Phonemized and Romanized Urdu for Text-to-Speech

Summary

Urdu has roughly a quarter of a billion speakers, yet its speech synthesis research is held back by small or unreleased corpora and by careless handling of pronunciation. Most earlier neural systems take Romanized Urdu as input, and that script is unstandardized and loses contrasts such as aspiration, retroflexion, nasalization and vowel length. The authors release URDUTTS, a studio-recorded corpus with single-speaker and multi-speaker subsets. Every utterance is transcribed three ways: native Urdu script, a Romanized version and a Phonemized version. They then train VITS and GlowTTS on the Romanized and the Phonemized inputs. The four systems are scored with objective metrics and with listening tests by native speakers. Phonemized input wins consistently on naturalness, intelligibility and acoustic distortion. A word-level error analysis shows that it handles retroflex distinctions better. The work supplies a sizeable open benchmark for Urdu TTS. It also gives empirical support for using phonetic rather than Latin-transliterated input in other phonologically rich, low-resource languages.

Key points

  • URDUTTS contains 89 hours of 22.05 kHz speech from four speakers in total. It is released under a permissive license at github.com/KAABSHAHID/URDUTTS.
  • The benchmark models are the non-autoregressive VITS and Glow-TTS, each trained once with Phonemized and once with Romanized transcriptions.
  • Objective evaluation used MCD (plain, DTW and slope-limited variants), SNR, WER and CER. Subjective evaluation used MOS ratings from 40 native Urdu listeners.
  • Switching to Phonemized input raised MOS by 9.6% for VITS and by 26.5% for GlowTTS.
  • In the linguistic analysis, errors cluster on short function words, retroflex consonants and rare, morphologically complex words. Three-character words had the lowest recognition accuracy. Romanized models confused the minimal pair "big"/"bad", which differs only by a retroflex sound.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Speech & spoken language Evaluation & benchmarks

Cite

Cite as M Kaab Bin Shahid and Muhammed Izharuddin. 2026. A Benchmark Dataset and Comparative Evaluation of Phonemized and Romanized Urdu for Text-to-Speech. Proceedings of the Fifteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2avnr98mgbre

@inproceedings{shahid2026benchmark,
  title = {{A Benchmark Dataset and Comparative Evaluation of Phonemized and Romanized Urdu for Text-to-Speech}},
  author = {M Kaab {Bin Shahid} and Muhammed Izharuddin},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2avnr98mgbre},
  url = {https://doi.org/10.63317/2avnr98mgbre}
}

References

36 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Farah Adeeba et al. 2016. Comparison of urdu text to speech synthesis using unit selection and hmm based techniques. In 2016 Conference of The Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques (O-COCOSDA), pages 1–6. IEEE. find
  2. Tafseer Ahmed. 2009. Roman to urdu transliteration using wordlist. In Proceedings of the Conference on Language and Technology, volume 305. ELRA. find
  3. M.S. Amin, X. Zhang, L. Anselma, and et al. 2025. Semantic processing for Urdu: corpus creation, parsing, and generation. Language Resources & Evaluation, 59:2469–2500. find
  4. S. Arif, A. J. Khan, M. Abbas, A. A. Raza, and A. Athar. 2025. WER we stand: Benchmarking Urdu ASR models. In Proceedings of the 31st International Conference on Computational Linguistics (COLING 2025), pages 5952– 5961, Abu Dhabi, UAE. Association for Computational Linguistics. find
  5. Elena Bashir. 2011. Urdu and linguistics: A fraught but evolving relationship. find
  6. Umer Butt, Stalin Veranasi, and Günter Neumann. 2025. Low-resource transliteration for roman-urdu and urdu using transformer-based models. arXiv preprint arXiv:2503.21530. find
  7. Coqui AI. 2020. Coqui tts: A deep learning toolkit for text-to-speech. https://github.com/coqui-ai/TTS. Accessed: 2025-10-09. eSpeak NG Project. 2015. espeak ng: An open source speech synthesizer. https://github.com/espeak-ng/espeak-ng. Accessed: 2025-10-09. ↗
  8. Tania Habib. 2014. Hidden markov model (hmm) based speech synthesis for urdu language. In Proceedings of the Conference on Language and Technology, Islamabad, Pakistan. Center for Language Engineering (CLE). find
  9. Wajiha Habib et al. 2014. Design of speech corpus for open domain urdu text-to-speech system using greedy algorithm. In Conference on Language and Technology (CLT14). find
  10. Ahmed Hanzala and Ayesha Kanwal. 2024. Generative urdu speech synthesis. In 2024 4th International Conference on Computer Systems (ICCS). IEEE. find
  11. Sarmad Hussain. 2004. Letter-to-sound conversion for urdu text-to-speech system. In Workshop on Computational Approaches to Arabic Script. find
  12. Sahar Jamal, Sadaf Abdul-Rauf, and Quratulain Majid. 2022. Exploring transfer learning for urdu speech synthesis. In Proceedings of the Workshop on Resources and Technologies for Indigenous, Endangered and Lesser-resourced Languages in Eurasia, 13th Language Resources and Evaluation Conference (LREC… find
  13. Tahir Javed, Kaushal Bhogale, Abhigyan Raman, Pratyush Kumar, Anoop Kunchukuttan, and Mitesh M. Khapra. 2023. Indicsuperb: a speech processing universal performance benchmark for indian languages. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Confer… find
  14. Tahir Javed, Janki Nawale, Eldho George, Sakshi Joshi, Kaushal Bhogale, Deovrat Mehendale, Ishvinder Sethi, Aparna Ananthanarayanan, Hafsah Faquih, Pratiti Palit, Sneha Ravishankar, Saranya Sukumaran, Tripura Panchagnula, Sunjay Murali, Kunal Gandhi, Ambujavalli R, Manickam M, Vaijayanthi C, Krishna… find
  15. Salman Ahmad Khan, Musadaq Mansoor, and Abdullah Habib. 2024. Overcoming linguistic barriers: Developing advanced urdu text-to-speech systems. In 2024 19th International Conference on Emerging Technologies (ICET). IEEE. find
  16. Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. 2020. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. In Advances in Neural Information Processing Systems (NeurIPS). find
  17. Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In Proceedings of the 38th International Conference on Machine Learning (ICML). find
  18. Robert Kubichek. 1993. Mel-cepstral distance measure for objective speech quality assessment. In Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing, volume 1, pages 125–128. IEEE. find
  19. Sébastien Le Maguer, Simon King, and Naomi Harte. 2024. The limits of the mean opinion score for speech synthesis evaluation. Computer Speech & Language, 84:101577. find
  20. Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, Ming Liu, and Ming Zhou. 2019. Neural speech synthesis with transformer network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6706–6713. find
  21. Y. Liu and et al. 2021. Delightfultts: The microsoft speech synthesis system for blizzard challenge. In Proceedings of the Blizzard Challenge Workshop. find
  22. Shivam Mehta, Jonas Külske, Matthias Sperber, Matthias Paulik, and Björn Schuller. 2022. Neural hmms are all you need (for high-quality attention-free tts). In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7897– 7901. IEEE. find
  23. Mozilla Foundation. 2020. Mozilla common voice: Urdu dataset. https://commonvoice.mozilla.org/en/datasets. Accessed: 2025-09-29. ↗
  24. Fatima Naseem et al. 2025. Developing high-quality tts for punjabi and urdu: Benchmarking against mms models. In Proceedings of Interspeech 2025. ISCA. find
  25. Sultan Mahmood Niazi and Perveen Akhter Farhat. 2023. Contrastive analysis of the vowel sound system of urdu and english languages. Pakistan Languages and Humanities Review, 7(4):228– 240. find
  26. Ryan Prenger, Rafael Valle, and Bryan Catanzaro. 2019. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617–3621. IEEE. find
  27. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proceedings of the International Conference o… our card
  28. Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2019. Fastspeech: Fast, robust and controllable text to speech. In Advances in Neural Information Processing Systems, volume 32, pages 3171– 3180. find
  29. Riffat Saba, Hikmat Ullah Khan, and Muhammad Bilal. 2022. Urdu text-to-speech conversion using deep learning. In 2022 International Conference on IT and Industrial Technologies (ICIT). IEEE. find
  30. Najm Ul Sehar et al. 2025. Benchmarking whisper for low-resource speech recognition: An n-shot evaluation on pashto, punjabi, and urdu. In Proceedings of the First Workshop on Challenges in Processing South Asian Languages (CHiP-SAL 2025). Association for Computational Linguistics. find
  31. Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan, and Rif A. Saurous. 2018. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Sp… find
  32. Keiichi Tokuda, Yoshihiko Nankaku, Tomoki Toda, Heiga Zen, Junichi Yamagishi, and Keiichiro Oura. 2013. Speech synthesis based on hidden markov models. In Proceedings of the IEEE, volume 101, pages 1234–1252. find
  33. Keiichi Tokuda, Takayoshi Yoshimura, Takashi Masuko, and Takao Kobayashi. 2000. Speech synthesis based on hidden markov models. Proceedings of the IEEE, 88(3):463–497. find
  34. M. Usama, S. Mehboob, and R. Waheed. 2024. Linguistic variations in urdu lexicon in everyday communication: A case study of the last decade of the modern century. Advances in Humanities Research, 4:25–36. find
  35. Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zhifeng Chen, Samy Bengio, Quoc V. Le, Yannis Agiomyrgiannakis, and Rob A. Saurous. 2017. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135. — [arXiv:1703.10135](https://arxiv.org/abs/1703.10135) find
  36. Wikipedia contributors. 2025. List of languages by total number of speakers. https://en.wikipedia.org/wiki/List_of_languages_by_total_number_of_speakers. [Online; accessed 29-September-2025]. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shahid-2026-benchmark-dataset-comparative-evaluation-phonemized/