Elamur Research⌕
☰
home / papers / 2026 / From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons

From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons

Paper · 2026 · Xiangyu Ma, Teng Xiao, Zuchao Li et al.

Summary

Diffusion language models can produce several tokens at once. However, they usually depend on bidirectional attention, which conflicts with how pre-trained autoregressive (AR) models such as GPT-style LLMs are built. Because of this, they are often trained from scratch at great cost. Existing block diffusion methods also use fixed-size blocks, which ignore how predictability varies across text. FLUID (Flexible Unidirectional Inference Diffusion) tackles both issues. First, it keeps a unidirectional attention mask during denoising ("Strictly Causal Alignment"), so an ordinary AR checkpoint can be fine-tuned directly and, in principle, keep KV caching. Second, it replaces fixed blocks with "Elastic Horizons": a K-Head estimates local entropy from hidden states and decides how many tokens to draft in each step. Training also adds mild noise to masked spans ("stochastic restoration") to make the model more robust. The authors evaluate FLUID on general understanding, math and code tasks. They report accuracy comparable to strong AR models, better results than bidirectional diffusion baselines, and training cost lower by orders of magnitude. This makes reusing existing AR models for parallel decoding practical.

Key points

  • Ablation results (GSM8K / MATH500 / HEval):
    • Bidirectional fixed-block baseline: 82.0 / 51.2 / 42.2.
    • Causal masking alone: 90.6 / 59.2 / 54.9.
    • Full FLUID (causal + elastic): 91.9 / 61.8 / 60.4.
  • Adaptive horizon length: the average horizon grows to K = 13.1 on GSM8K and shrinks to K = 6.5 on the harder MMLU. Horizon length correlates negatively with uncertainty.
  • Throughput on MMLU: FLUID reaches 18.82 tokens/s, compared with 17.52 when large fixed strides are forced, because it spends less effort on error correction.
  • Best hyperparameters:
    • Stochastic restoration noise: 10% works best; 0% and 15% are both worse.
    • Competence boundary: τ = 2.8 is optimal, with a bell-shaped trend around it.
  • Release and limitations: code is published at huggingface.co/MYTH-Lab/FLUID. Experiments mainly adapt general-purpose LLMs such as OpenPangu, so FLUID inherits the base model's weaknesses. Specialized domains and MoE architectures have not yet been tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Wuhan UniversityWuhan · China35 works · 1,127 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Xiangyu Ma, Teng Xiao, Zuchao Li and Lefei Zhang. 2026. From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.958

@inproceedings{ma2026ar,
  title = {{From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons}},
  author = {Xiangyu Ma, Teng Xiao, Zuchao Li and Lefei Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.958},
  url = {https://doi.org/10.18653/v1/2026.acl-long.958}
}

References

42 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. 2025. Block diffusion: Interpolating between autoregressive and diffusion language models. In The Thirteenth International Conference on Learning Representations. find
  2. Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. 2021a. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981–17993.
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021b. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  4. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, and 67 others. 2024. Deepseek LLM: scaling open-source language models with… find
  5. Hanting Chen, Yasheng Wang, Kai Han, Dong Li, Lin Li, Zhenni Bi, Jinpeng Li, Haoyu Wang, Fei Mi, Mingjian Zhu, and 1 others. 2025a. Pangu embedded: An efficient dual-system llm reasoner with metacognition. arXiv preprint arXiv:2505.22375.
  6. Jinglin Chen, Qiwei Li, Zuchao Li, Baoyuan Qi, Liu Guoming, Haojun Ai, Hai Zhao, and Ping Wang. 2025b. Faster in-context learning for LLMs via ngram trie speculative decoding. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 18040–18051, Suzhou, China.… find
  7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  8. Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, Kai Chen, Biqing Qi, and 1 others. 2025. Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation. arXiv preprint arXiv:2510.06303. find
  9. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. Qwen2-audio technical report. CoRR, abs/2407.10759.
  10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168.
  11. DeepCtrl. 2024. deepctrl-sft-data. https://www.modelscope.cn/datasets/deepctrl/ deepctrl-sft-data. ↗
  12. Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. 2024. Language modeling is compression. In The Twelfth International Conference on Learning R…
  13. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3029–3051, Si…
  14. Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. 2025. Scaling diffusion language models via adaptation from autoregressive models. In The Thirteenth International Conference on Learning Representat… find
  15. Ishaan Gulrajani and Tatsunori B. Hashimoto. 2023. Likelihood-based diffusion language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. find
  16. Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. 2023a. Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11575– 11596. find
  17. Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. 2023b. SSD-LM: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11575– 11596,… find
  18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  19. Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, and 1 others. 2025. Mercury: Ultra-fast language models based on diffusion. arXiv preprint arXiv:2506.17298, 1. find
  20. Jijie Li, Li Du, Hanyu Zhao, Bo wen Zhang, Liangdong Wang, Boyan Gao, Guang Liu, and Yonghua Lin. 2025. Infinity instruct: Scaling instruction selection and synthesis to enhance language models. Preprint, arXiv:2506.11116. — [arXiv:2506.11116](https://arxiv.org/abs/2506.11116) find
  21. Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. 2022. Diffusion-lm improves controllable text generation. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA,…
  22. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  23. Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, and 1 others. 2025a. Skywork-reward-v2: Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352.
  24. Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. 2025b. dllm-cache: Accelerating diffusion large language models with adaptive caching. CoRR, abs/2506.06295. — [arXiv:2506.06295](https://arxiv.org/abs/2506.06295) find
  25. Aaron Lou, Chenlin Meng, and Stefano Ermon. 2024. Discrete diffusion language modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning (ICML). find
  26. Shi Luohe, Zuchao Li, Lefei Zhang, Baoyuan Qi, Liu Guoming, and Hai Zhao. 2025. KV-latent: Dimensional-level KV cache reduction with frequency-aware rotary positional embedding. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages…
  27. Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. 2025. dkv-cache: The cache for diffusion language models. CoRR, abs/2505.15781. — [arXiv:2505.15781](https://arxiv.org/abs/2505.15781) find
  28. Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. 2024. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514. — [arXiv:2410.18514](https://arxiv.org/abs/2410.18514) find
  29. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, JUN ZHOU, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. Large language diffusion models. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy.
  30. Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, and 1 others. 2024. Moss: An open conversational large language model. Machine Intelligence Research, 21(5):888–905. find
  31. Zicong Tang, Shi Luohe, Zuchao Li, Baoyuan Qi, Liu Guoming, Lefei Zhang, and Ping Wang. 2025. SpindleKV: A novel KV cache reduction method balancing both shallow and deep layers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages…
  32. Llama Team. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.
  33. Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, Yonggan Fu, Zhijian Liu, Pavlo Molchanov, Ping Luo, Song Han, and Enze Xie. 2025a. Fast-dllm v2: Efficient block-diffusion llm. arXiv preprint arXiv:2509.26328. — [arXiv:2509.26328](https://arxiv.org/abs/2509.26328) find
  34. Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. 2025b. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618. — [arXiv:2505.22618](https://arxiv.org/abs/2505.22618) find
  35. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024. Qwen2.5 technical report. CoRR, abs/2412.1511… find
  36. Haoqi Yang, Yao Yao, Zuchao Li, Baoyuan Qi, Liu Guoming, and Hai Zhao. 2025. XQuant: Achieving ultra-low bit KV cache quantization with cross-layer compression. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9785–9800, Suzhou, China. Association for… find
  37. Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. 2025a. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487. find
  38. Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. 2025b. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487. find
  39. Yiming Zeng, Jinghan Cao, Zexin Li, Yiming Chen, Tao Ren, Dawei Xiang, Xidong Wu, Shangqian Gao, and Tingting Yu. 2025. Treediff: Ast-guided code generation with diffusion llms. arXiv preprint arXiv:2508.01473.
  40. Yi Zhao, Zuchao Li, and Hai Zhao. 2025. IAM: Efficient inference through attention mapping between different-scale LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 19522– 19533, Vienna, Austria. Association for Computatio… find
  41. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.
  42. Kun Zhou, Yifan Li, Wayne Xin Zhao, and Ji-Rong Wen. 2024. Diffusion-nat: Self-prompting discrete diffusion for non-autoregressive text generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1438–1451. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2026-ar-diffusion-efficiently-adapting-large/