Elamur Research⌕
☰
home / papers / 2023 / AudioLDM: Text-to-Audio Generation with Latent Diffusion Models

AudioLDM: Text-to-Audio Generation with Latent Diffusion Models

Paper · 2023 · Haohe Liu, Zehua Chen, Yi Yuan et al.

Summary

Text-to-audio (TTA) generation aims to produce general sound, such as effects, music or speech, from free-form text. Earlier systems like DiffSound and AudioGen produced limited quality at high compute cost. They also depended on scarce, noisy audio-caption pairs, which they often reduced to label-like text, losing temporal and spatial detail. AudioLDM moves the problem into a continuous latent space. A VAE compresses mel-spectrograms, and a latent diffusion model generates these latents, with a vocoder producing the waveform. The diffusion model is conditioned on CLAP embeddings, a contrastively trained joint audio–text space. During training it sees only the embedding of the audio itself, and at inference the CLAP text embedding takes its place, so captions are not needed to train the generator. On AudioCaps, with training on a single GPU, AudioLDM reaches an FD of 23.31 against 47.68 for DiffSound, leading open-source systems on both objective and subjective scores. The same model also performs zero-shot text-guided manipulations such as style transfer, super-resolution and inpainting.

Key points

  • Pipeline:
    • mel-spectrogram, then a VAE latent compressed by factor r, then a conditional LDM in that latent space;
    • VAE decoder plus vocoder to get audio;
    • CLAP conditioning uses a RoBERTa text encoder trained with a symmetric cross-entropy loss.
  • Headline result: FD 23.31 on AudioCaps, compared with 47.68 for the DiffSound baseline. The authors report that audio-only LDM training even beats training on audio–text pairs.
  • Compression level (AudioLDM-S):
    • FD is 29.48, 33.50 and 34.32 for r = 4, 8 and 16.
    • Even r = 16, which keeps 4 frequency dimensions out of 64 mel bands, still beats DiffSound on every metric.
    • r = 1 or 2 was impractical on one RTX 3090, so r = 4 is the default.
  • Zero-shot manipulation:
    • Super-resolution (8 kHz to 16 kHz) on AudioCaps and VCTK, scored with LSD, beats AudioUNet but trails NVSR.
    • Inpainting refills the 2.5–7.5 s gap and is scored with FAD.
  • Ablations:
    • Simplifying UNet attention hurts all metrics.
    • Balanced sampling brings no gain.
    • Conditioning augmentation raises subjective OVL/REL scores but not objective ones.
    • Moving from 100 to 200 DDIM steps adds only a small improvement.

Interesting ideas

  • I1. Swap conditions through a shared embedding space. The generator is trained on embeddings of the target modality and queried with the paired modality at test time. This removes the need for large paired datasets and could transfer to any domain with a contrastive encoder but few aligned pairs.
  • I2. Preserve rich descriptions instead of collapsing them into labels. The paper argues that turning captions into label-like strings throws away relational and temporal information that matters for generation. This is a useful warning for any text-preprocessing pipeline.
  • I3. Get editing tasks for free from the reverse diffusion process. Inpainting, super-resolution and style transfer come from partially noising an input and denoising it under a new text condition, with no task-specific fine-tuning. This makes a single generative model a cheap baseline for many restoration tasks.
  • I4. Treat compression level as an explicit, measured trade-off. Sweeping r shows how much quality is lost per gain in compute and memory. This is a good template for choosing latent sizes under a single-GPU budget.
  • I5. Report subjective and objective metrics side by side. Conditioning augmentation helped human ratings while hurting dataset-matched metrics, a reminder that augmentation can push outputs away from the benchmark distribution without making them worse to listeners.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksSignal Processing · 493 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 519 works

Cite

Cite as Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang and Mark D. Plumbley. 2023. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. arXiv:2301.12503.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.12503 · arXiv 2301.12503

@misc{liu2023audioldm,
  title = {{AudioLDM: Text-to-Audio Generation with Latent Diffusion Models}},
  author = {Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang and Mark D. Plumbley},
  year = {2023},
  doi = {10.48550/arxiv.2301.12503},
  eprint = {2301.12503},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.12503}
}

References

63 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andresen, U. A new way in sound synthesis. In Audio Engineering Society. Audio Engineering Society, 1979. find
  2. Chen, K., Du, X., Zhu, B., Ma, Z., Berg-Kirkpatrick, T., and Dubnov, S. HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2022a. find
  3. Chen, N., Zhang, Y., Zen, H., Weiss, R., Norouzi, M., and Chan, W. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021. find
  4. Chen, Z., Tan, X., Wang, K., Pan, S., Mandic, D., He, L., and Zhao, S. Infergrad: Improving diffusion models for vocoder by considering inference in training. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2022b. find
  5. Chen, Z., Wu, Y., Leng, Y., Chen, J., Liu, H., Tan, X., Cui, Y., Wang, K., He, L., Zhao, S., Bian, J., and Mandic, D. Resgrad: Residual denoising diffusion probabilistic models for text to speech. arXiv preprint:2212.14518, 2022c. — [arXiv:2212.14518](https://arxiv.org/abs/2212.14518) find
  6. Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. In Conference on Neural Information Processing Systems, 2021. find
  7. Drossos, K., Lipping, S., and Virtanen, T. Clotho: an audio captioning dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2020. find
  8. Engel, J., Hantrakul, L., Gu, C., and Roberts, A. Ddsp: Differentiable digital signal processing. arXiv preprint:2001.04643, 2020. — [arXiv:2001.04643](https://arxiv.org/abs/2001.04643) find
  9. Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. AudioSet: An ontology and human-labeled dataset for audio events. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 776–780. IEEE, 2017. find
  10. Gong, Y., Chung, Y.-A., and Glass, J. PSLA: Improving audio tagging with pretraining, sampling, labeling, and aggregation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3292–3306, 2021. find
  11. Hershey, S., Chaudhuri, S., Ellis, D. P., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. CNN architectures for large-scale audio classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 131–135. find
  12. Ho, J. and Salimans, T. Classifier-free diffusion guidance. In NeurIPS Workshop on Deep Generative Models and Downstream Applications, 2021.
  13. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Conference on Neural Information Processing Systems, 2020. find
  14. Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., and Salimans, T. Imagen video: High definition video generation with diffusion models. arXiv preprint:2210.02303, 2022. — [arXiv:2210.02303](https://arxiv.org/abs/2210.02303) find
  15. Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1125–1134, 2017. find
  16. Karplus, K. and Strong, A. Digital synthesis of plucked-string and drum timbres. Computer Music Journal, 7(2): 43–55, 1983. find
  17. Kilgour, K., Zuluaga, M., Roblek, D., and Sharifi, M. Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms. In INTERSPEECH, pp. 2350–2354, 2019. find
  18. Kim, C. D., Kim, B., Lee, H., and Kim, G. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 119–132, 2019.
  19. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint:1412.6980, 2014.
  20. Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint:1312.6114, 2013.
  21. Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022–17033, 2020a.
  22. Kong, Q., Cao, Y., Iqbal, T., Wang, Y., Wang, W., and Plumbley, M. D. PANNs: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880–2894, 2020b. find
  23. Kong, Q., Cao, Y., Liu, H., Choi, K., and Wang, Y. Decoupling magnitude and phase estimation with deep resunet for music source separation. arXiv preprint:2109.05418, 2021a. — [arXiv:2109.05418](https://arxiv.org/abs/2109.05418) find
  24. Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021b. find
  25. Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., Défossez, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y. Audiogen: Textually guided audio generation. arXiv preprint:2209.15352, 2022. find
  26. Kuleshov, V., Enam, S. Z., and Ermon, S. Audio super resolution using neural networks. arXiv:1708.00853, 2017. find
  27. Lam, M., Wang, J., Huang, R., Su, D., and Yu, D. Bilateral denoising diffusion models. In International Conference on Learning Representations, 2022. find
  28. Lee, S., Kim, H., Shin, C., Tan, X., Liu, C., Meng, Q., Qin, T., Chen, W., Yoon, S., and Liu, T. Priorgrad: Improving conditional denoising diffusion models with data-driven adaptive prior. In International Conference on Learning Representations, 2022. find
  29. Leng, Y., Chen, Z., Guo, J., Liu, H., Chen, J., Tan, X., Mandic, D., He, L., Li, X.-Y., Qin, T., et al. Binauralgrad: A two-stage conditional diffusion probabilistic model for binaural audio synthesis. arXiv preprint:2205.14807, 2022. — [arXiv:2205.14807](https://arxiv.org/abs/2205.14807) find
  30. Liu, H., Kong, Q., Tian, Q., Zhao, Y., Wang, D., Huang, C., and Wang, Y. Voicefixer: Toward general speech restoration with neural vocoder. arXiv preprint:2109.13731, 2021a. — [arXiv:2109.13731](https://arxiv.org/abs/2109.13731) find
  31. Liu, H., Choi, W., Liu, X., Kong, Q., Tian, Q., and Wang, D. Neural vocoder is all you need for speech super-resolution. arXiv preprint:2203.14941, 2022a. — [arXiv:2203.14941](https://arxiv.org/abs/2203.14941) find
  32. Liu, H., Kong, Q., Liu, X., Mei, X., Wang, W., and Plumbley, M. D. Ontology-aware learning and evaluation for audio tagging. arXiv preprint:2211.12195, 2022b. — [arXiv:2211.12195](https://arxiv.org/abs/2211.12195) find
  33. Liu, H., Liu, X., Kong, Q., Wang, W., and Plumbley, M. D. Learning the spectrogram temporal resolution for audio classification. arXiv preprint:2210.01719, 2022c. — [arXiv:2210.01719](https://arxiv.org/abs/2210.01719) find
  34. Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, J. B. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, 2022d. find
  35. Liu, X., Iqbal, T., Zhao, J., Huang, Q., Plumbley, M. D., and Wang, W. Conditional sound generation using neural discrete time-frequency representation learning. In IEEE International Workshop on Machine Learning for Signal Processing, pp. 1–6. IEEE, 2021b. find
  36. Liu, X., Liu, H., Kong, Q., Mei, X., Plumbley, M. D., and Wang, W. Simple pooling front-ends for efficient audio classification. arXiv preprint:2210.00943, 2022e. — [arXiv:2210.00943](https://arxiv.org/abs/2210.00943) find
  37. Liu, X., Liu, H., Kong, Q., Mei, X., Zhao, J., Huang, Q., Plumbley, M. D., and Wang, W. Separate what you describe: language-queried audio source separation. arXiv preprint:2203.15147, 2022f. — [arXiv:2203.15147](https://arxiv.org/abs/2203.15147) find
  38. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint:1907.11692, 2019.
  39. Nichol, A. and Dhariwal, P. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, 2021. find
  40. Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint:2112.10741, 2021. find
  41. Oord, A. v. d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. WaveNet: A generative model for raw audio. arXiv preprint:1609.03499, 2016.
  42. Pascual, S., Bhattacharya, G., Yeh, C., Pons, J., and Serrà, J. Full-band general audio synthesis with score-based diffusion. arXiv preprint:2210.14661, 2022. — [arXiv:2210.14661](https://arxiv.org/abs/2210.14661) find
  43. Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  44. Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudinov, M. Grad-tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, 2021. find
  45. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
  46. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020.
  47. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint:2204.06125, 2022.
  48. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022.
  49. Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M. Image super-resolution via iterative refinement. arXiv preprint:2104.07636, 2021. — [arXiv:2104.07636](https://arxiv.org/abs/2104.07636) find
  50. Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint:2205.11487, 2022.
  51. Salamon, J., Jacoby, C., and Bello, J. P. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 1041–1044, 2014. find
  52. Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint:2111.02114, 2021.
  53. Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., Parikh, D., Gupta, S., and Taigman, Y. Make-a-video: Text-to-video generation without text-video data. arXiv preprint:2209.14792, 2022. find
  54. Sinha, A., Song, J., Meng, C., and Ermon, S. D2c: Diffusion-decoding models for few-shot conditional generation. In Conference on Neural Information Processing Systems, 2021. find
  55. Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint:2010.02502, 2020.
  56. Song, Y., Sohl-Dickstein, J., Kingma, D., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  57. Tan, X., Chen, J., Liu, H., Cong, J., Zhang, C., Liu, Y., Wang, X., Leng, Y., Yi, Y., He, L., et al. NaturalSpeech: End-to-end text to speech synthesis with human-level quality. arXiv preprint:2205.04421, 2022. find
  58. Vahdat, A., Kreis, K., and Kautz, J. Lsgm: Score-based generative modeling in latent space. In Conference on Neural Information Processing Systems, 2021. find
  59. Wang, H. and Wang, D. Towards robust speech super-resolution. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2058–2066, 2021. find
  60. Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., and Dubnov, S. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. arXiv preprint:2211:06687, 2022. find
  61. Yamagishi, J., Veaux, C., MacDonald, K., et al. CSTR VCTK corpus: English multi-speaker corpus for cstr voice cloning toolkit. 2019. find
  62. Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., and Yu, D. Diffsound: Discrete diffusion model for text-to-sound generation. arXiv preprint:2207.09983, 2022. — [arXiv:2207.09983](https://arxiv.org/abs/2207.09983) find
  63. Żelaszczyk, M. and Mańdziuk, J. Audio-to-image cross-modal generation. In International Joint Conference on Neural Networks, pp. 1–8. IEEE, 2022. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2023-audioldm-text-audio-generation-latent/