Elamur Research⌕
☰
home / papers / 2025 / Phir Hera Fairy: An English Fairytaler is a Strong Faker of Fluent Speech in Low-Resource Indian Languages

Phir Hera Fairy: An English Fairytaler is a Strong Faker of Fluent Speech in Low-Resource Indian Languages

Paper · 2025 · Praveen Srinivasa Varadhan, Srija Anand, Soma Siddhartha et al.

Summary

The paper asks whether a large English-only TTS model can be a cheap starting point for speech synthesis in low-resource Indian languages, avoiding the cost of collecting huge datasets and training from scratch. The authors take F5-TTS, a flow-matching model pretrained on about 100K hours of English, and add raw native-script characters to its vocabulary, so no grapheme-to-phoneme step is needed. They compare three regimes on a 1417-hour, 11-language corpus (IN11): training from scratch, fine-tuning on Indian data only, and fine-tuning on Indian plus English data. Indian-only fine-tuning works best. The resulting IN-F5 model approaches human quality, can make a speaker of one Indian language sound fluent in another, and handles code-mixed speech. The authors also vary fine-tuning data from 1 to 100 hours per language to find a compute-efficient recipe. Finally, they bootstrap TTS for Bhojpuri and Tulu, which have no training data, using synthetic speech checked by native speakers. The work shows that English pretraining is a strong prior for underserved languages.

Key points

  • Base model: English F5-TTS. The released adapted model is IN-F5 (IndicF5 on Hugging Face). Its vocabulary has 685 character tokens covering the IN11 scripts.
  • IN11 spans 1417 hours across eleven languages: Hindi, Marathi, Gujarati, Punjabi, Bengali, Assamese, Odia, Tamil, Telugu, Kannada and Malayalam.
    • Sources: IndicTTS, LIMMITS, Rasa, Google Crowdsourced TTS and the ASR-restored IndicVoices-R.
    • The IN11-Test-Set holds 1100 utterances, balanced across seen and unseen speakers, gender and age.
  • Training ran up to 150K steps with AdamW at learning rate 5e-5 on 32 H100 GPUs. The fine-tuning data is roughly one hundredth of the English pretraining volume.
  • Evaluation combines several measures:
    • MUSHRA naturalness (Nat), plus two new anchored scales, MUSHRA-S for speaker similarity (Sim) and MUSHRA-I for intelligibility (Int).
    • 134 listeners took part, about 12 native raters per language.
    • Objective metrics are WER from the Indic-Conformer ASR model and WavLM cosine similarity between real and synthesized speech.
  • Zero-resource extension:
    • IN-F5 generates 1 hour of synthetic speech per new language, using a Kannada voice for Tulu and a Maithili voice for Bhojpuri.
    • Native speakers verify every sample, and the model is then self-trained on the verified set.
    • The final models score MUSHRA 82.0 for Bhojpuri and 93.6 for Tulu.

Interesting ideas

  • I1. When new tokens are added to a pretrained model, the authors initialize their embeddings by sampling from embeddings the model already has, which keeps new rows inside the learned space and speeds convergence.
  • I2. Shared script lets a model trained on a related language (Kannada for Tulu, Hindi for Bhojpuri) produce first drafts for a new one, and human verification turns those drafts into training data.
  • I3. They pretend a language with real recordings (Bhojpuri) is zero-resource, so the synthetic-data pipeline can be scored against genuine human data.
  • I4. Each subjective scale gets explicitly anchored levels (100, 80, 60… with a description for each), and separate scales cover separate properties instead of one overall quality score.
  • I5. Test data is split by seen versus unseen speakers, so generalization is measured rather than memorization.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): When the tagger or lemma models gain new vocabulary items (rare symbols, new affixes), seed their vectors from existing in-distribution vectors rather than random noise, so fine-tuning is shorter and more stable.
  • I2 (learn): Use the current parser on unseen English registers or dialects to produce silver trees, keep only those a checker or LLM teacher tier accepts, and retrain on them as a controlled self-training loop.
  • I3 (learn): Deliberately hide an existing gold treebank or QA set, run the silver-data pipeline as if it did not exist, then measure against the hidden gold to calibrate how much to trust silver data elsewhere.
  • I4 (answer): Grade generated answers and explanations on several anchored rubrics, such as correctness, faithfulness to the text world and clarity, so human or LLM judges give comparable, diagnosable scores.
  • I5 (learn): Split evaluation sets by seen and unseen authors, sources or story templates, so parser and reader gains on familiar material are not mistaken for real generalization.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology MadrasChennai · India16 works · 2,499 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSpeech & spoken language · 480 worksLow-resource languages · 267 works

Cite

Cite as Praveen Srinivasa Varadhan, Srija Anand, Soma Siddhartha and Mitesh M.Khapra. 2025. Phir Hera Fairy: An English Fairytaler is a Strong Faker of Fluent Speech in Low-Resource Indian Languages. arXiv:2505.20693.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.20693 · arXiv 2505.20693

@misc{varadhan2025phir,
  title = {{Phir Hera Fairy: An English Fairytaler is a Strong Faker of Fluent Speech in Low-Resource Indian Languages}},
  author = {Praveen Srinivasa Varadhan, Srija Anand, Soma Siddhartha and Mitesh M.Khapra},
  year = {2025},
  doi = {10.48550/arxiv.2505.20693},
  eprint = {2505.20693},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.20693}
}

References

24 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Wang, H. Zhan, L. Liu, R. Zeng, H. Guo, J. Zheng, Q. Zhang, S. Zhang, and Z. Wu, “Maskgct: Zero-shot text-to-speech with masked generative codec transformer,” CoRR, vol. abs/2409.00750, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2409.00750 ↗
  2. X. Tan, J. Chen, H. Liu, J. Cong, C. Zhang, Y. Liu, X. Wang, Y. Leng, Y. Yi, L. He, S. Zhao, T. Qin, F. K. Soong, and T. Liu, “Naturalspeech: End-to-end text-to-speech synthesis with human-level quality,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 6, pp. 4234–4245, 2024. [Online]. Available: https://doi.org/10.1109/TPAMI.2024.3356232 ↗
  3. P. Anastassiou, J. Chen, J. Chen, Y. Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gao et al., “Seed-tts: A family of high-quality versatile speech generation models,” arXiv preprint arXiv:2406.02430, 2024. ↗
  4. K. Shen, Z. Ju, X. Tan, E. Liu, Y. Leng, L. He, T. Qin, sheng zhao, and J. Bian, “Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=Rc 7dAwVL3v ↗
  5. P. Peng, P. Huang, S. Li, A. Mohamed, and D. Harwath, “Voicecraft: Zero-shot speech editing and text-to-speech in the wild,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, L. Ku, A. Martins, and V. Srikumar, Eds. Association for Computational Linguistics, 2024, pp. 12 442–12 462. [Online]. Available: https://doi.org/10.18653/v1/2024.acl-long.673 ↗
  6. Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, and M. Tagliasacchi, “Soundstorm: Efficient parallel audio generation,” arXiv preprint arXiv:2305.09636, 2023. ↗
  7. Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y. Liu, Y. Leng, K. Song, S. Tang, Z. Wu, T. Qin, X.-Y. Li, W. Ye, S. Zhang, J. Bian, L. He, J. Li, and S. Zhao, “Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” 2024. ↗
  8. M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar et al., “Voicebox: Text-guided multilingual universal speech generation at scale,” Advances in neural information processing systems, vol. 36, 2024. find
  9. Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang et al., “Cosyvoice 2: Scalable streaming speech synthesis with large language models,” arXiv preprint arXiv:2412.10117, 2024. ↗
  10. Y. Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. Zhao, K. Yu, and X. Chen, “F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,” arXiv preprint arXiv:2410.06885, 2024. ↗
  11. E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. Gölge, and M. A. Ponti, “Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,” in International Conference on Machine Learning. PMLR, 2022, pp. 2709–2720. ↗
  12. G. K. Kumar, V. PraveenS., P. Kumar, M. M. Khapra, and K. Nandakumar, “Towards building text-to-speech systems for the next billion users,” ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:253581696 ↗
  13. P. S. Varadhan, A. Gulati, A. Sankar, S. Anand, A. Gupta, A. Mukherjee, S. K. Marepally, A. Bhatia, S. Jaju, S. Bhooshan et al., “Rethinking mushra: Addressing modern challenges in text-to-speech evaluation,” arXiv preprint arXiv:2411.12719, 2024. ↗
  14. S. Anand, P. Srinivasa Varadhan, A. Sankar, G. Raju, and M. M. Khapra, “Enhancing out-of-vocabulary performance of indian tts systems for practical applications through low-effort data strategies,” in Interspeech 2024, 2024, pp. 1200–1204. find
  15. H. He, Z. Shang, C. Wang, X. Li, Y. Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi et al., “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,” in 2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, pp. 885–890. find
  16. A. Sankar, S. Anand, P. S. Varadhan, S. Thomas, M. Singal, S. Kumar, D. Mehendale, A. Krishana, G. Raju, and M. M. Khapra, “Indicvoices-r: Unlocking a massive multilingual multi-speaker speech corpus for scaling indian TTS,” NeurIPS 2024 Datasets and Benchmarks track, Vancouver, Canada, vol. abs/2409.05356, 2024. [Online]. Available: https://doi.org/10.4 8550/arXiv.2409.05356 ↗
  17. A. Singh, A. Nagireddi, D. G, J. Bandekar, R. R, S. Badiger, S. Udupa, P. K. Ghosh, H. A. Murthy, P. Kumar, K. Tokuda, M. Hasegawa-Johnson, and P. Olbrich, “Limmits’24: Multi-speaker, multi-lingual indic tts with voice cloning,” in 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), 2024, pp. 61–62. find
  18. M. Narayanan and N. Aepli, “A Tulu resource for machine translation,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue, Eds. Torino, Italia: ELRA and ICCL, May 2024, pp. 1756–1767. [Online]. Available: https://aclanthology.org/2024.lrec-main.155/ ↗
  19. P. S. Varadhan, A. Sankar, G. Raju, and M. M. Khapra, “Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settings,” in Proc. INTERSPEECH 2024, 2024. find
  20. A. Baby, A. L. Thomas, N. Nishanthi, T. Consortium et al., “Resources for indian languages,” in Proceedings of Text, Speech and Dialogue, 2016. find
  21. B. Abraham, D. Goel, D. Siddarth, K. Bali, M. Chopra, M. Choudhury, P. Joshi, P. Jyoti, S. Sitaram, and V. Seshadri, “Crowdsourcing speech data for low-resource languages from low-income workers,” in Proceedings of the 12th Language Resources and Evaluation Conference, 2020, pp. 2819–2826. find
  22. T. Javed, J. A. Nawale, E. I. George, S. Joshi, K. S. Bhogale, D. Mehendale, I. V. Sethi, A. Ananthanarayanan, H. Faquih, P. Palit et al., “Indicvoices: Towards building an inclusive multilingual speech dataset for indian languages,” arXiv preprint arXiv:2403.01926, 2024. ↗
  23. S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE J. Sel. Top. Signal Process., vol. 16, no. 6, pp. 1505–1518, 2022. [Online]. Available: https://doi.org/10.1109/JSTSP.2022.3188113 ↗
  24. A. Prakash, S. Umesh, and H. A. Murthy, “Towards developing state-of-the-art TTS synthesisers for 13 indian languages with signal processing aided alignments,” in IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2023, Taipei, Taiwan, December 16-20, 2023. IEEE, 2023, pp. 1–8. [Online]. Available: https://doi.org/10.1109/ASRU57964.2023.10389630 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/varadhan-2025-phir-hera-fairy-english-fairytaler/