Elamur Research⌕
☰
home / papers / 2026 / One-step Nonautoregressive Natural Language Generation with Shortcut Flow Matching Models

One-step Nonautoregressive Natural Language Generation with Shortcut Flow Matching Models

Paper · 2026 · Jędrzej Warczyński, Ondrej Dusek and Mateusz Lango

Summary

Diffusion-style non-autoregressive (NAR) text generators can in principle decode all tokens in parallel. In practice they need many denoising iterations, and their quality collapses when the iteration count is cut drastically. Flow-matching models also lose much of their quality when restricted to one step. The authors bring shortcut models, previously developed for images and later applied to speech, into text generation. A BERT-base-sized Transformer encoder is conditioned on the step size as well as the time variable. Training mixes three objectives:

  • the usual flow-matching velocity loss;
  • a self-consistency term, which requires one double-length jump to agree with two consecutive half-length jumps;
  • a rounding loss, which keeps token embeddings from collapsing.

At inference the model makes a single jump from noise to the output. They evaluated it on three paraphrasing benchmarks (QQP, PAWS-Wiki, ParaSCI) with BLEU, BERTScore, ROUGE and the LLM-based Themis. The shortcut variant clearly outperforms one-step flow matching and roughly doubles BLEU on two of the datasets. This narrows the gap to diffusion systems that need 2000 steps, which matters for fast parallel decoding.

Key points

  • One-step BLEU, baseline vs. shortcut:
    • QQP: 14.20 → 19.40 (BERTScore 0.617 → 0.814)
    • PAWS-Wiki: 20.97 → 42.71
    • ParaSCI: 6.05 → 13.55
    • All gains are significant (t-test, p < 0.001).
  • On QQP, DiffuSeq (31.49 BLEU) and SeqDiffuSeq (26.67 BLEU) still score higher, but they use 2000 denoising steps versus one.
  • Adding decoding steps (up to 256) barely helps the shortcut model. The baseline improves with more steps but never reaches the shortcut model's one-step quality.
  • Five extensions were tested on QQP:
    • Self-conditioning and full BERT initialization mainly speed up convergence (BERT initialization converges in about 10k steps).
    • CFG and embedding-only initialization bring no gain, and frozen BERT embeddings fail entirely (BLEU 0.0).
  • Themis scores (fluency, semantic similarity, overall quality) improve on all datasets, e.g. overall quality on PAWS-Wiki rises from 1.03 to 2.02, though absolute quality remains modest. Code is released at github.com/jwarczynski/FMShortcut.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Charles UniversityPrague · Czechia133 works · 20,686 citations🎓Poznań University of TechnologyPoznan · Poland3 works · 1 citation

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Text generation · 202 works

Cite

Cite as Jędrzej Warczyński, Ondrej Dusek and Mateusz Lango. 2026. One-step Nonautoregressive Natural Language Generation with Shortcut Flow Matching Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.53

@inproceedings{warczynski2026one,
  title = {{One-step Nonautoregressive Natural Language Generation with Shortcut Flow Matching Models}},
  author = {Jędrzej Warczyński, Ondrej Dusek and Mateusz Lango},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.53},
  url = {https://doi.org/10.18653/v1/2026.acl-short.53}
}

References

31 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain. Association for Computational Linguistics.
  2. DataCanary, hilfialkaff, Lili Jiang, Meg Risdal, Nikhil Dandekar, and tomtung. 2017. Quora question pairs. https://kaggle.com/competitions/ quora-question-pairs. Kaggle. ↗
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  4. Qingxiu Dong, Xiaojun Wan, and Yue Cao. 2021. ParaSCI: A large scientific paraphrase dataset for longer paraphrase generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 424–434, Online. Association for Computati… find
  5. Betty Fabre, Tanguy Urvoy, Jonathan Chevelu, and Damien Lolive. 2021. Neural-driven search-based paraphrase generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2100–2111, Online. Association for Computational… find
  6. Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. 2025. One step diffusion via shortcut models. In The Thirteenth International Conference on Learning Representations. find
  7. Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2023. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations. find
  8. Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. 2023. DiffusionBERT: Improving generative masked language models with diffusion models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4521…
  9. Jonathan Ho and Tim Salimans. 2022. Classifier-free diffusion guidance. Preprint, arXiv:2207.12598.
  10. Vincent Hu, Di Wu, Yuki Asano, Pascal Mettes, Basura Fernando, Björn Ommer, and Cees Snoek. 2024a. Flow matching for conditional text generation in a few sampling steps. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Pa… find
  11. Xinyu Hu, Li Lin, Mingqi Gao, Xunjian Yin, and Xiaojun Wan. 2024b. Themis: A reference-free NLG evaluation language model with flexibility and interpretability. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15924–15951, Miami, Florida, USA. Associat… find
  12. Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. 2024. Impossible distillation for paraphrasing and summarization: How to make high-quality lemonade out of small, low-quality model. In Proceedings of the 2024 Conference of the North Ame… find
  13. Sangyun Lee, Beomsu Kim, and Jong Chul Ye. 2023. Minimizing trajectory curvature of ODE-based generative models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 18957–18973. PMLR. find
  14. Yifan Li, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Diffusion models for non-autoregressive text generation: A survey. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 6692–6701. International Joint Conferences on Artificial Intell… find
  15. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  16. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2023. Flow matching for generative modeling. Preprint, arXiv:2210.02747.
  17. Pan Liu, Xiaohua Tian, and Zhouhan Lin. 2024. Enable fast sampling for Seq2Seq text diffusion. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8495–8505, Miami, Florida, USA. Association for Computational Linguistics. find
  18. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations (ICLR 2019). Also available as arXiv:1711.05101. find
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  20. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1).
  21. Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, and Rémi Leblond. 2022. Self-conditioned embedding diffusion for text generation. Preprint, arXiv:2211.04236. — [arXiv:2211.04236](https://arxiv.org/abs/2211.04236) find
  22. Jaesung Tae, Hamish Ivison, Sachin Kumar, and Arman Cohan. 2025. TESS 2: A large-scale generalist diffusion language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 21171–21188, Vienna, Austria. Association for Computat…
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  24. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  25. Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. 2025. Energy-based diffusion language models for text generation. In International Conference on Learning Representations (ICLR). Also available as arXiv:2410.21357. find
  26. Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. 2024. Text diffusion model with encoder-decoder transformers for sequence-to-sequence generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
  27. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  28. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308, M…
  29. Xiaochen Zhu, Georgi Karadzhov, Chenxi Whitehouse, and Andreas Vlachos. 2025. Segment-level diffusion: A framework for controllable long-form generation with diffusion language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  30. Wei Zou, Ziyuan Zhuang, Xiang Geng, Shujian Huang, Jia Liu, and Jiajun Chen. 2024. Improved paraphrase generation via controllable latent diffusion. arXiv preprint arXiv:2404.08938. find
  31. Jialong Zuo, Shengpeng Ji, Minghui Fang, Mingze Li, Ziyue Jiang, Xize Cheng, Xiaoda Yang, Chen Feiyang, Xinyu Duan, and Zhou Zhao. 2025. Rhythm controllable and efficient zero-shot voice conversion via shortcut flow matching. In Proceedings of the 63rd Annual Meeting of the Association for Computati… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/warczynski-2026-one-step-nonautoregressive-natural-language/