Elamur Research
☰
home / papers / 2019 / LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech

LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1904.02882

Summary

LibriSpeech is large, multi-speaker and openly licensed, so TTS researchers often use it. It was built for speech recognition, though, which causes problems for synthesis. Its audio is only 16 kHz. Utterances are cut at pauses rather than sentence ends. Its transcripts are uppercase without punctuation, and it drops paragraph context. Its "clean" portion can still be noisy. The authors went back to the original LibriVox MP3s and Project Gutenberg texts and rebuilt the corpus specifically for TTS, keeping LibriSpeech's speakers, subset split and permissive license.

Their pipeline:

  • segments the book text into paragraphs and sentences and normalizes it;
  • uses ASR to locate the text for each chapter;
  • force-aligns the audio to sentences with a modified version of YouTube's auto-sync engine;
  • discards misaligned, overly long or noisy segments.

The resulting LibriTTS corpus provides 24 kHz audio with sentence-level segmentation, both raw and normalized text, and access to neighbouring sentences. Neural end-to-end TTS models trained on it reached a naturalness MOS above 4.0 for most test speakers. This makes it a useful open benchmark for multi-speaker and prosody research.

Key points

  • Size and license: LibriTTS contains about 585 hours of speech from 2,456 speakers at 24 kHz. It is released under CC-BY 4.0 at openslr.org/60. Files below 24 kHz were dropped.
  • Subsets: It mirrors LibriSpeech's subsets: train-clean-100, train-clean-360, train-other-500, and the dev and test clean/other sets. Gender balance is roughly even (1,185 male and 1,271 female speakers).
  • Alignment: Chapter text was located with Google Cloud Speech-to-Text. Audio was then aligned by decoding with a sentence-specific tri-gram mini-LM and a BLSTM acoustic model. Only sentences matching with zero edit distance were kept.
  • Filtering: Sentences longer than 71 words were removed, as were segments with suspiciously long average word duration. Noise filtering used WADA-SNR, with thresholds of 20 dB for "clean" subsets and 0 dB for "other" subsets.
  • Trade-off: The stricter filtering makes per-speaker duration more uneven. The median per speaker is about 900 s, compared with about 1,500 s in LibriSpeech. In TTS baselines, models still scored above 4.0 MOS for five of six evaluation speakers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

128 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 124 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.02882 · arXiv 1904.02882

@misc{zen2019libritts,
  title = {{LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech}},
  author = {Zen, Heiga and Dang, Viet and Clark, Rob and Zhang, Yu and Weiss, Ron J. and Jia, Ye and Chen, Zhifeng and Wu, Yonghui},
  year = {2019},
  doi = {10.48550/arxiv.1904.02882},
  eprint = {1904.02882},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.02882}
}

References

47 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Sotelo, S. Mehri, K. Kumar, J. Santos, K. Kastner, A. Courville, and Y. Bengio, “Char2Wav: End-to-End speech synthesis,” in Proc. ICLR workshop, 2017. find
  2. S. Arik, C. Chrzanowski, A. Coates, G. Diamos, A. Gibiansky, Y. Kang, X. Li et al., “Deep Voice: Real-time neural text-to-speech,” in Proc. ICML, 2017, pp. 195–204. find
  3. Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. Weiss, N. Jaitly, Z. Yang et al., “Tacotron: A fully end-to-end text-to-speech synthesis model,” in Proc. Interspeech, 2017, pp. 4006–4010. find
  4. J. Shen, R. Pang, J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen et al., “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,” in Proc. ICASSP, 2018, pp. 4779–4783. find
  5. N. Li, S. Liu, Y. Liu, S. Zhao, M. Liu, and M. Zhou, “Close to human quality TTS with transformer,” arXiv:1809.08895, 2018. — [arXiv:1809.08895](https://arxiv.org/abs/1809.08895) find
  6. S. Arik, G. Diamos, A. Gibiansky, J. Miller, K. Peng, W. Ping, J. Raiman et al., “Deep Voice 2: Multi-speaker neural text-to-speech,” in Proc. NIPS, 2017, pp. 2962–2970. find
  7. W. Ping, K. Peng, A. Gibiansky, S. Arik, A. Kannan, S. Narang, J. Raiman et al., “Deep Voice 3: 2000-speaker neural text-to-speech,” in Proc. ICLR, 2018. find
  8. Y. Jia, Y. Zhang, R. Weiss, Q. Wang, J. Shen, F. Ren, Z. Chen et al., “Transfer learning from speaker verification to multispeaker text-to-speech synthesis,” arxiv:1806.04558, 2018. — [arXiv:1806.04558](https://arxiv.org/abs/1806.04558) find
  9. Y.-A. Chung, Y. Wang, W.-N. Hsu, Y. Zhang, and R. Skerry-Ryan, “Semi-supervised training for improving data efficiency in end-to-end speech synthesis,” arXiv:1808.10128, 2018. — [arXiv:1808.10128](https://arxiv.org/abs/1808.10128) find
  10. S. Arik, J. Chen, P. W. Peng, Kainan and, and Y. Zhou, “Neural voice cloning with a few samples,” arXiv:1802.06006, 2018. — [arXiv:1802.06006](https://arxiv.org/abs/1802.06006) find
  11. Y. Chen, Y. Assael, B. Shillingford, D. Budden, S. Reed, H. Zen, A. Wang et al., “Sample efficient adaptive text-to-speech,” arXiv:1809.10460, 2018. — [arXiv:1809.10460](https://arxiv.org/abs/1809.10460) find
  12. Y. Wang, D. Stanton, Y. Zhang, R. Skerry-Ryan, E. Bettenberg, J. Shor, Y. Xiao et al., “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,” in Proc. ICML, 2018, pp. 5167–5176. find
  13. W.-N. Hsu, Y. Zhang, R. Weiss, H. Zen, Y. Wu, Y. Wang, Y. Cao et al., “Hierarchical generative modeling for controllable speech synthesis,” arXiv:1810.07217, 2018. — [arXiv:1810.07217](https://arxiv.org/abs/1810.07217) find
  14. Y. Taigman, L. Wolf, A. Polyak, and E. Nachmani, “Voice synthesis for in-the-wild speakers via a phonological loop,” arxiv:1707.06588, 2017. — [arXiv:1707.06588](https://arxiv.org/abs/1707.06588) find
  15. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “LibriSpeech: an ASR corpus based on public domain audio books,” in Proc. ICASSP, 2015, pp. 5206–5210. find
  16. “LibriVox – Free public domain audiobooks,” https://librivox. org/. find
  17. T. Capes, P. Coles, A. Conkie, L. Golipour, A. Hadjitarkhani, Q. Hu, N. Huddleston et al., “Siri on-device deep learning-guided unit selection text-to-speech system,” in Proc. Interspeech, 2017, pp. 4011–4015. find
  18. A. van den Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. van den Driessche et al., “Parallel WaveNet: Fast high-fidelity speech synthesis,” arXiv:1711.10433, 2017. — [arXiv:1711.10433](https://arxiv.org/abs/1711.10433) find
  19. J. Kominek and A. Black, “CMU ARCTIC databases for speech synthesis,” Carnegie Mellon University, Tech. Rep. CMU-LTI-03-177, 2003. find
  20. C. Veaux, J. Yamagishi, and K. MacDonald, “CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit,” University of Edinburgh. The Centre for Speech Technology Research (CSTR) http://dx.doi.org/10.7488/ds/1994, 2012. — [doi:10.7488/ds/1994](https://doi.org/10.7488/ds/1994) ↗
  21. “Open Data Commons Attribution License (ODC-By) v1.0,” https://opendatacommons.org/files/2018/02/odc\_by\_1.0\_public\_text.txt. ↗
  22. S. King and V. Karaiskos, “The Blizzard Challenge 2011,” in Blizzard Challenge Workshop, 2011. find
  23. ——, “The Blizzard Challenge 2013,” in Blizzard Challenge Workshop, 2013. https://keithito.com/ ↗
  24. K. Ito, “The LJ speech dataset,” LJ-Speech-Dataset/, 2017. find
  25. “CC0 1.0 Universal Public Domain Dedication,” https://creativecommons.org/publicdomain/zero/1.0/. ↗
  26. Munich Artificial Intelligence Laboratories GmbH, “The M-AILABS speech dataset,” https://www.caito.de/2019/01/the-m-ailabs-speech-dataset/. ↗
  27. “Creative Commons Attribution 4.0 License (CC-BY 4.0),” https: //creativecommons.org/licenses/by/4.0/. find
  28. H. Zen, K. Tokuda, and A. Black, “Statistical parametric speech synthesis,” Speech Commn., vol. 51, no. 11, pp. 1039–1064, 2009. find
  29. E. Nachmani, A. Polyak, Y. Taigman, and L. Wolf, “Fitting new speakers based on a short untranscribed sample,” arXiv;1802.06984, 2018. find
  30. Y. Lee, T. Kim, and S. Lee, “Voice imitating text-to-speech neural networks,” arXiv:1806.00927, 2018. — [arXiv:1806.00927](https://arxiv.org/abs/1806.00927) find
  31. D.-R. Liu, C.-Y. Yang, S.-L. Wu, and H.-Y. Lee, “Improving unsupervised style transfer in end-to-end speech synthesis with end-to-end speech recognition,” in Proc. SLT, 2018, pp. 640–647. find
  32. R. Skerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang, D. Stanton, J. Shor, R. J. Weiss et al., “Towards end-to-end prosody transfer for expressive speech synthesis with Tacotron,” in Proc. ICML, 2018, pp. 4700–4709. find
  33. D. Stanton, Y. Wang, and R. Skerry-Ryan, “Predicting expressive speaking style from text in end-to-end speech synthesis,” arxiv:1808.01410, 2018. — [arXiv:1808.01410](https://arxiv.org/abs/1808.01410) find
  34. K. Kastner, J. F. Santos, Y. Bengio, and A. C. Courville, “Representation mixing for TTS synthesis,” arXiv:1811.07240, 2018. — [arXiv:1811.07240](https://arxiv.org/abs/1811.07240) find
  35. “GitHub – mozilla/TTS: Deep learning for Text to Speech,” https: //github.com/mozilla/TTS. find
  36. P. Taylor, Text-to-speech synthesis. 2009. Cambridge university press, find
  37. P. Ebden and R. Sproat, “The Kestrel TTS text normalization system,” Journal of Natural Language Engineering, vol. 21, no. 3, pp. 333–353, 2015. find
  38. https://cloud.google.com/speech-to-text/. ↗
  39. C. Alberti and M. Bacchiani, “Automatic captioning in YouTube,” https://ai.googleblog.com/2009/12/automatic-captioning-in-youtube.html, 2009. ↗
  40. H. Liao, E. McDermott, and A. Senior, “Large scale deep neural network acoustic modeling with semi-supervised training data for YouTube video transcription,” in Proc. ASRU, 2013, pp. 368–373. find
  41. P. J. Moreno and C. Alberti, “A factor automaton approach for the forced alignment of long speech recordings,” in Proc. ICASSP, 2009, pp. 4869–4872. find
  42. H. Sak, A. Senior, and F. Beaufays, “Long short-term memory recurrent neural network architectures for large scale acoustic modeling,” in Proc. Interspeech, 2014, pp. 338–342. find
  43. C. Kim and R. M. Stern, “Robust signal-to-noise ratio estimation based on waveform amplitude distribution analysis,” in Proc. Interspeech, 2009, pp. 2598–2561. find
  44. J. L. Hintze and R. D. Nelson, “Violin plots: A box plot-density trace synergism,” The American Statistician, vol. 52, no. 2, pp. 181–184, 1998. find
  45. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv:1412.6980, 2014. find
  46. N. Kalchbrenner, E. Erich, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg et al., “Efficient neural audio synthesis,” arXiv:1802.08435, 2018. find
  47. H. Hofman, K. Kafadar, and H. Wickham, “Letter-value plots: Boxplots for large data,” had.co.nz, Tech. Rep., 2011. Table 5: The frequencies of characters in train subsets. character frequency character frequency find