Elamur Research⌕
☰
home / papers / 2023 / DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Models

DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Models

Summary

Seq2Seq text diffusion models such as DiffuSeq produce good text, but they train slowly and generate slowly. They also usually rely on Minimum Bayes Risk (MBR) decoding over several samples to reach good quality. DiffuSeq-v2 speeds up both training and generation by combining continuous and discrete noise.

During training, the model first adds Gaussian noise in embedding space. It then randomly swaps some target-token vectors for a learned "soft absorbing state", a trainable vector that plays a role similar to a [MASK] token. The swap probability is scaled with the timestep, and the model learns to remove both kinds of corruption together. The same discrete noise is applied at sampling time, which brings training and inference closer together. Because of this, the clamping step is no longer needed and the DPM-solver++ ODE sampler can be used.

On the QQP paraphrase benchmark, the method converges more than 4× faster in total training time. Without MBR, it matches the quality of DiffuSeq with MBR. A 2-step variant generates about 800× faster than single-sample DiffuSeq while scoring higher. This moves diffusion-based text generation closer to practical use.

Key points

  • Training speed: FP16 alone cut training from 28 to 11 hours on 2 A100 GPUs (about 2.5×). The soft absorbing state made convergence at least 1.75× faster, for more than 4× overall.
  • Quality without MBR: On QQP, the full 2000-step version reaches BLEU 0.2411, close to DiffuSeq with MBR=10 (0.2413).
  • 2-step speedup: The 2-step DPM-solver++ version scores BLEU 0.2115 versus 0.1884 for DiffuSeq (MBR=1), a 12.3% relative gain. Its throughput is 406.5 it/s versus 0.51 it/s, roughly 800× faster.
  • Other baselines: The method beats the continuous baseline BG-DiffuSeq. Compared with discrete models (Multinomial, D3PM-absorbing), it remains competitive and has a clear advantage in generation speed. Reparameterized discrete variants (RDM) were not directly compared, and RDM+absorbing reaches higher BLEU than the 2-step version.
  • Ablations:
    • Removing clamping costs DiffuSeq 3.4% BLEU but the new model only 0.5%.
    • Leaving out the discrete noise at sampling time cuts BLEU by 14.9%.
    • A [MASK] ratio of γ = 0.5 worked best.
  • Limitations: The method was tested only on QQP, with no translation or summarization experiments and no study of larger models. Code is available in the diffuseq-v2 branch of Shark-NLP/DiffuSeq.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.660 · arXiv 2310.05793

@inproceedings{gong2023diffuseq,
  title = {{DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Models}},
  author = {Gong, Shansan and Li, Mukai and Feng, Jiangtao and Wu, Zhiyong and Kong, Lingpeng},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.660},
  eprint = {2310.05793},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.660}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. 2021. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems. find our card
  2. Jiaao Chen, Aston Zhang, Mu Li, Alex Smola, and Diyi Yang. 2023. A cheaper and better diffusion language model with soft-masked noise. ArXiv, abs/2304.04746. find
  3. Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. 2022. Analog bits: Generating discrete data using diffusion models with self-conditioning. ArXiv, abs/2208.04202. — [arXiv:2208.04202](https://arxiv.org/abs/2208.04202) find
  4. Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. 2022. Difformer: Empowering diffusion model on embedding space for text generation. ArXiv, abs/2212.09412. — [arXiv:2212.09412](https://arxiv.org/abs/2212.09412) find
  5. Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2023. DiffuSeq: Sequence to sequence text generation with diffusion models. In International Conference on Learning Representations, ICLR. find
  6. Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. 2023. Efficient diffusion training via min-snr weighting strategy. ArXiv, abs/2303.09556. — [arXiv:2303.09556](https://arxiv.org/abs/2303.09556) find
  7. Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. 2022. Diffusionbert: Improving generative masked language models with diffusion models. ArXiv, abs/2211.15029. find
  8. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems. find
  9. Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. 2021. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems. find
  10. Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang, Zuopeng Yang, Dacheng Tao, and Ponnuthurai N Suganthan. 2022. Unified discrete diffusion for simultaneous vision-language generation. ArXiv, abs/2211.14842. — [arXiv:2211.14842](https://arxiv.org/abs/2211.14842) find
  11. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 388–395. find
  12. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… find our card
  13. Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto. 2022. Diffusion-lm improves controllable text generation. ArXiv, abs/2205.14217. find our card
  14. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. find
  15. Zheng-Wen Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. 2022. Text generation with diffusion language models: A pre-training approach with continuous paragraph denoise. volume abs/2212.11685. — [arXiv:2212.11685](https://arxiv.org/abs/2212.11685) find
  16. Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongx-uan Li, and Jun Zhu. 2022a. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Conference on Neural Information Processing Systems, NeurIPS. find
  17. Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongx-uan Li, and Jun Zhu. 2022b. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. ArXiv, abs/2211.01095. — [arXiv:2211.01095](https://arxiv.org/abs/2211.01095) find
  18. Rabeeh Karimi Mahabadi, Jaesung Tae, Hamish Ivison, James Henderson, Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2023. Tess: Text-to-text self-conditioned simplex diffusion. ArXiv, abs/2305.08379. find
  19. Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, ICML. find
  20. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1–9. Association for Computational Linguistics. find our card
  21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL. find our card
  22. Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. In International Conference on Learning Representations, ICLR. find our card
  23. Zecheng Tang, Pinzheng Wang, Keyan Zhou, Juntao Li, Ziqiang Cao, and M. Zhang. 2023. Can diffusion model achieve better performance in text generation? bridging the gap between training and inference! ArXiv, abs/2305.04465. find our card
  24. Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. 2022. Seqdiffuseq: Text diffusion with encoder-decoder transformers. ArXiv, abs/2212.10325. — [arXiv:2212.10325](https://arxiv.org/abs/2212.10325) find
  25. Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023. Diffusum: Generation enhanced extractive summarization with diffusion. ArXiv, abs/2305.01735. find
  26. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, ICLR. find
  27. Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. 2023. A reparameterized discrete diffusion model for text generation. ArXiv, abs/2302.05737. — [arXiv:2302.05737](https://arxiv.org/abs/2302.05737) find
  28. Kun Zhou, Yifan Li, Wayne Xin Zhao, and Ji rong Wen. 2023. Diffusion-nat: Self-prompting discrete diffusion for non-autoregressive text generation. ArXiv, abs/2305.04044. find
  29. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1097–1100. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/gong-2023-diffuseq-v2-bridging-discrete-continuous/