Elamur Research⌕
☰
home / papers / 2018 / WaveGlow: A Flow-based Generative Network for Speech Synthesis

WaveGlow: A Flow-based Generative Network for Speech Synthesis

Paper · 2018 · Ryan Prenger, Rafael Valle and Bryan Catanzaro

Summary

Neural vocoders convert time-aligned acoustic features such as mel-spectrograms into waveforms. The strongest ones, like WaveNet, generate audio one sample at a time, so they make poor use of GPUs. Existing parallel alternatives (Parallel WaveNet, ClariNet, MCNN) are fast, but they depend on teacher–student distillation or several combined losses, which makes them hard to train and reproduce. The authors propose WaveGlow, a vocoder built as a normalizing flow. It maps Gaussian noise to audio through a stack of invertible steps. Each step pairs a Glow-style invertible 1x1 convolution with an affine coupling layer, whose scale and shift come from a non-causal, WaveNet-like network conditioned on the upsampled mel-spectrogram. Because every step can be inverted, the exact data likelihood is maximized directly, using one network and one loss. The model was trained on the LJ speech dataset. The PyTorch implementation generates audio at over 500 kHz on a V100 GPU, more than 25× faster than real time. In Mean Opinion Score tests it matches a strong open-source WaveNet. This makes high-quality non-autoregressive synthesis much easier to deploy.

Key points

  • Data and features: The model is trained on the LJ speech dataset, about 13,100 clips (roughly 24 hours) of one speaker reading non-fiction, sampled at 22,050 Hz. Inputs are 80-bin mel-spectrograms with FFT size 1024, hop 256 and window 1024.
  • Architecture: Audio is "squeezed" into vectors of 8 samples, then passed through 12 steps of flow, each an invertible 1x1 convolution plus an affine coupling layer. Each coupling sub-network (WN) has 8 dilated convolution layers with 3 taps, gated-tanh units, 512 residual channels and 256 skip channels.
  • Training: Training ran on 8 GV100 GPUs with random 16,000-sample clips for 580,000 iterations. It used Adam, weight normalization, batch size 24 and learning rate 1×10⁻⁴.
  • Baselines: WaveGlow is compared with Griffin-Lim (60 iterations) and with Ryuichi Yamamoto's open-source WaveNet vocoder (24 layers, trained for 10⁶ iterations).
  • Results: Synthesis runs above 500 kHz on an NVIDIA V100, and MOS quality matches the best publicly available WaveNet trained on the same data.

Interesting ideas

  • I1. Invertibility without an invertible sub-network: In a coupling layer, half of the channels pass through unchanged and drive an arbitrary network that rescales and shifts the other half. Any expressive architecture can therefore sit inside an exactly invertible model, which applies to density modeling in any domain.
  • I2. One likelihood loss instead of distillation: Replacing a teacher–student setup and hand-tuned auxiliary losses with plain maximum likelihood made training simple and stable. When a pipeline is fragile to reproduce, a design with a single exact objective is often worth the trade-off.
  • I3. Learnable channel mixing that stays invertible: A 1x1 convolution initialized as an orthonormal matrix mixes channels between coupling layers. Its log-determinant enters the loss, which keeps it invertible during training. This is a cheap general-purpose way to add learned permutations to flow models.
  • I4. Early outputs as a gradient shortcut: Every 4 coupling layers, 2 channels are sent straight to the loss. This lets the model capture structure at different depths and helps gradients reach early layers, similar to skip connections, and it could help other deep stacked generative models.
  • I5. Sampling with reduced temperature: At inference, noise is drawn with a smaller standard deviation (0.6) than the one assumed in training (√0.5 ≈ 0.71), which slightly improved audio quality. This is a free quality knob for any likelihood-trained generator.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢NvidiaSanta Clara · United States37 works · 12,656 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 519 works

Cite

Cite as Ryan Prenger, Rafael Valle and Bryan Catanzaro. 2018. WaveGlow: A Flow-based Generative Network for Speech Synthesis. arXiv:1811.00002.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1811.00002 · arXiv 1811.00002

@misc{prenger2018waveglow,
  title = {{WaveGlow: A Flow-based Generative Network for Speech Synthesis}},
  author = {Ryan Prenger, Rafael Valle and Bryan Catanzaro},
  year = {2018},
  doi = {10.48550/arxiv.1811.00002},
  eprint = {1811.00002},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.00002}
}

References

22 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Diederik P Kingma and Prafulla Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” arXiv preprint arXiv:1807.03039, 2018. find
  2. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu, “Wavenet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499, 2016.
  3. Ryan Prenger and Rafael Valle, “Waveglow,” https://nv-adlr.github.io/WaveGlow, 2018. ↗
  4. Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al., “Tacotron: A fully end-to-end text-to-speech synthesis model,” arXiv preprint arXiv:1703.10135, 2017. find
  5. Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan, et al., “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,” arXiv preprint arXiv:1712.05884, 2017. find
  6. Sercan O Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, et al., “Deep voice: Real-time neural text-to-speech,” arXiv preprint arXiv:1702.07825, 2017. find
  7. Wei Ping, Kainan Peng, and Jitong Chen, “Clarinet: Parallel wave generation in end-to-end text-to-speech,” arXiv preprint arXiv:1807.07281, 2018. — [arXiv:1807.07281](https://arxiv.org/abs/1807.07281) find
  8. Sercan O Arik, Heewoo Jun, and Gregory Diamos, “Fast spectrogram inversion using multi-head convolutional neural networks,” arXiv preprint arXiv:1808.06719, 2018. — [arXiv:1808.06719](https://arxiv.org/abs/1808.06719) find
  9. Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, To… find
  10. Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling, “Improved variational inference with inverse autoregressive flow,” in Advances in Neural Information Processing Systems, 2016, pp. 4743–4751. find
  11. Laurent Dinh, David Krueger, and Yoshua Bengio, “Nice: Non-linear independent components estimation,” arXiv preprint arXiv:1410.8516, 2014. — [arXiv:1410.8516](https://arxiv.org/abs/1410.8516) find
  12. Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio, “Density estimation using real nvp,” arXiv preprint arXiv:1605.08803, 2016. — [arXiv:1605.08803](https://arxiv.org/abs/1605.08803) find
  13. Danilo Jimenez Rezende and Shakir Mohamed, “Variational inference with normalizing flows,” arXiv preprint arXiv:1505.05770, 2015.
  14. Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, and Alexander Ku, “Image transformer,” arXiv preprint arXiv:1802.05751, 2018. find
  15. Keith Ito et al., “The LJ speech dataset,” 2017. find
  16. Daniel Griffin and Jae Lim, “Signal estimation from modified short-time fourier transform,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 32, no. 2, pp. 236–243, 1984. find
  17. Ryuichi Yamamoto, “Wavenet vocoder,” https://doi.org/10.5281/zenodo.1472609, 2018. — [doi:10.5281/zenodo.1472609](https://doi.org/10.5281/zenodo.1472609) ↗
  18. Diederik P Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  19. Tim Salimans and Diederik P Kingma, “Weight normalization: A simple reparameterization to accelerate training of deep neural networks,” in Advances in Neural Information Processing Systems, 2016, pp. 901–909. find
  20. Sercan Arik, Gregory Diamos, Andrew Gibiansky, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou, “Deep voice 2: Multi-speaker neural text-to-speech,” arXiv preprint arXiv:1705.08947, 2017. find
  21. Zeyu Jin, Adam Finkelstein, Gautham J. Mysore, and Jingwan Lu, “FFTNet: a real-time speaker-dependent neural vocoder,” in The 43rd IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Apr. 2018. find
  22. Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu, “Efficient neural audio synthesis,” arXiv preprint arXiv:1802.08435, 2018. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/prenger-2018-waveglow-flow-based-generative-network/