Elamur Research⌕
☰
home / papers / 2026 / Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing

Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing

Paper · 2026 · Lingkun Long, Yushi Huang, Shihao Bai et al.

Summary

Diffusion LLMs (dLLMs) decode by iteratively denoising a masked sequence with bidirectional full attention, which becomes very expensive for long inputs. Sparse attention could help, but it usually needs the query tokens that are about to be decoded, and in dLLMs those positions are not known in advance. Focus-dLLM is a training-free framework that addresses this gap. The authors observe that a token's confidence at one denoising step strongly predicts its confidence at the next. They use the previous step's confidence to guess which positions will be unmasked, then widen these guesses into local windows. These predicted queries select only the most relevant key-value pairs. Attention sinks are always kept, and their locations are reused across deeper layers because sinks stay consistent between layers. Shallow layers keep full attention. Built on approximate KV caching, the method matches or exceeds full-attention accuracy on LongBench while sharply speeding up long-context inference. This makes long-input dLLM use far more practical.

Key points

  • Predicting unmasked positions: on LLaDA-8B-Instruct with GSM8K, the four highest-confidence remasked tokens at step t−1 recover the tokens decoded at step t with 96.1% average recall.
  • Scaling with context length: speedup over vanilla decoding rises from 9.4× at 8K to 29.6× at 32K, with generation length and steps fixed at 256.
  • Comparison with Fast-dLLM: at 32K, Focus-dLLM is up to 2.05× faster than Fast-dLLM on UltraLLaDA.
  • Ablation on UltraLLaDA (LongBench, 16K):
    • Fast-dLLM alone scores 44.74 average at 11.03 throughput.
    • The full method, combining PCGI with sink-aware sparse attention, scores 45.14 at 17.71 throughput.
  • Effect of attention sinks: keeping sinks raises the LongBench average on Dream-7B-Instruct from 41.47 to 42.82. Code is public at github.com/Longxmas/Focus-dLLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations🎓Beihang UniversityBeijing · China65 works · 8,452 citations🏢SensetimeBeijing · China12 works · 2,065 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Lingkun Long, Yushi Huang, Shihao Bai, Ruihao Gong, Jun Zhang, Ao Zhou and Jianlei Yang. 2026. Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.556

@inproceedings{long2026focus,
  title = {{Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing}},
  author = {Lingkun Long, Yushi Huang, Shihao Bai, Ruihao Gong, Jun Zhang, Ao Zhou and Jianlei Yang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.556},
  url = {https://doi.org/10.18653/v1/2026.acl-long.556}
}

References

59 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. 2025. Block diffusion: Interpolating between autoregressive and diffusion language models. Preprint, arXiv:2503.09573. — [arXiv:2503.09573](https://arxiv.org/abs/2503.09573) find
  2. Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. 2021. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981–17993.
  3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, and 29 others. 2023. Qwen technical report. Preprint, arXiv:2309.16609.
  4. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, and 1 others. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd annual meeting of the association for computational lin…
  5. Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu, Zenan Huang, Zhenzhong Lan, Chengxi Li, Chongxuan Li, Jianguo Li, Zehuan Li, Huabin Liu, Lin Liu, Guoshan Lu, Xiaocheng Lu, Yuxin Ma, and 12 others. 2025. Llada2.0: Scaling up diffusion language models to 1… — [arXiv:2512.15745](https://arxiv.org/abs/2512.15745) find
  6. Sitong Chen, Shen Nie, Jiacheng Sun, Zijin Feng, Zhenguo Li, Ji-Rong Wen, and Chongxuan Li. 2025. Masked diffusion models as energy minimization. arXiv preprint arXiv:2509.13866. — [arXiv:2509.13866](https://arxiv.org/abs/2509.13866) find
  7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  8. OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/ opencompass. ↗
  9. Xingtong Ge, Yi Zhang, Yushi Huang, Dailan He, Xia-hong Wang, Bingqi Ma, Guanglu Song, Yu Liu, and Jun Zhang. 2026. Salt: Self-consistent distribution matching with cache-aware training for fast video generation. arXiv preprint arXiv:2604.03118. — [arXiv:2604.03118](https://arxiv.org/abs/2604.03118) find
  10. Ruihao Gong, Yang Yong, Shiqiao Gu, Yushi Huang, Chengtao Lv, Yunchen Zhang, Dacheng Tao, and Xianglong Liu. 2024. Llmc: Benchmarking large language model quantization with a versatile compression toolkit. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Ind… find
  11. Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. 2025. Scaling diffusion language models via adaptation from autoregressive models. Preprint, arXiv:2410.17891. — [arXiv:2410.17891](https://arxiv.org/abs/2410.17891) find
  12. Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. 2022. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933. find
  13. Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. 2025. When attention sink emerges in language models: An empirical view. Preprint, arXiv:2410.10781. — [arXiv:2410.10781](https://arxiv.org/abs/2410.10781) find
  14. Guangxin He, Shen Nie, Fengqi Zhu, Yuankang Zhao, Tianyi Bai, Ran Yan, Jie Fu, Chongxuan Li, and Binhang Yuan. 2025. Ultrallada: Scaling the context length to 128k for diffusion large language models. Preprint, arXiv:2510.10481. — [arXiv:2510.10481](https://arxiv.org/abs/2510.10481) find
  15. Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuan-Jing Huang, and Xipeng Qiu. 2023. Diffusionbert: Improving generative masked language models with diffusion models. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers), pages 452…
  16. Jianuo Huang, Yaojie Zhang, Yicun Yang, Benhao Huang, Biqing Qi, Dongrui Liu, and Linfeng Zhang. 2025a. Mask tokens as prophet: Fine-grained cache eviction for efficient dllm inference. Preprint, arXiv:2510.09309. find
  17. Yushi Huang, Xingtong Ge, Ruihao Gong, Chengtao Lv, and Jun Zhang. 2025b. Linvideo: A post-training framework towards o (n) attention in efficient video generation. arXiv preprint arXiv:2510.08318. — [arXiv:2510.08318](https://arxiv.org/abs/2510.08318) find
  18. Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, and Xianglong Liu. 2024. Tfmq-dm: Temporal feature maintenance quantization for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7362– 7371. find
  19. Yushi Huang, Ruihao Gong, Jing Liu, Yifu Ding, Chengtao Lv, Haotong Qin, and Jun Zhang. 2025c. Qvgen: Pushing the limit of quantized video generative models. arXiv preprint arXiv:2505.11497. find
  20. Yushi Huang, Ruihao Gong, Xianglong Liu, Jing Liu, Yuhang Li, Jiwen Lu, and Dacheng Tao. 2025d. Temporal feature matters: A framework for diffusion model quantization. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  21. Yushi Huang, Zining Wang, Ruihao Gong, Jing Liu, Xinjie Zhang, Jinyang Guo, Xianglong Liu, and Jun Zhang. 2025e. Harmonica: Harmonizing training and inference for better feature caching in diffusion transformer acceleration. In International Conference on Machine Learning, pages 25835–25858. PMLR. find
  22. Yushi Huang, Zining Wang, Zhihang Yuan, Yifu Ding, Ruihao Gong, Jinyang Guo, Xianglong Liu, and Jun Zhang. 2025f. Modes: Accelerating mixture-of-experts multimodal large language models via dynamic expert skipping. arXiv preprint arXiv:2511.15690. — [arXiv:2511.15690](https://arxiv.org/abs/2511.15690) find
  23. Yuchu Jiang, Yue Cai, Xiangzhong Luo, Jiale Fu, Jiarui Wang, Chonghan Liu, and Xu Yang. 2025. d 2 cache: Accelerating diffusion-based llms via dual adaptive caching. Preprint, arXiv:2509.23094. — [arXiv:2509.23094](https://arxiv.org/abs/2509.23094) find
  24. Xiaoran Liu, Yuerong Song, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. 2025a. Longllada: Unlocking long context capabilities in diffusion llms. Preprint, arXiv:2506.14429. — [arXiv:2506.14429](https://arxiv.org/abs/2506.14429) find
  25. Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. 2025b. dllm-cache: Accelerating diffusion large language models with adaptive caching. Preprint, arXiv:2506.06295. — [arXiv:2506.06295](https://arxiv.org/abs/2506.06295) find
  26. Lingkun Long, Rubing Yang, Yushi Huang, Desheng Hui, Ao Zhou, and Jianlei Yang. 2026. Sliminfer: Accelerating long-context llm inference via dynamic token pruning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 32284–32292. find
  27. Aaron Lou, Chenlin Meng, and Stefano Ermon. 2023. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834. find
  28. Chengtao Lv, Yumeng Shi, Yushi Huang, Ruihao Gong, Shen Ren, and Wenya Wang. 2026a. Light forcing: Accelerating autoregressive video diffusion via sparse attention. arXiv preprint arXiv:2602.04789. find
  29. Chengtao Lv, Bilang Zhang, Yang Yong, Ruihao Gong, Yushi Huang, Shiqiao Gu, Jiajun Wu, Yumeng Shi, Jinyang Guo, and Wenya Wang. 2026b. Llmc+: Benchmarking vision-language model compression with a plug-and-play toolkit. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, page… find
  30. Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. 2025. dkv-cache: The cache for diffusion language models. Preprint, arXiv:2505.15781. — [arXiv:2505.15781](https://arxiv.org/abs/2505.15781) find
  31. Quan Nguyen-Tri, Mukul Ranjan, and Zhiqiang Shen. 2025. Attention is all you need for kv cache in diffusion llms. Preprint, arXiv:2510.14973. — [arXiv:2510.14973](https://arxiv.org/abs/2510.14973) find
  32. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. Large language diffusion models. Preprint, arXiv:2502.09992.
  33. G. Kamradt. 2023. Needle in a haystack - pressure testing llms. https://github.com/gkamradt/ LLMTest_NeedleInAHaystack. ↗
  34. Maximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri, Simone Scardapane, and Alessio Devoto. 2025. Attention sinks in diffusion language models. Preprint, arXiv:2510.15731.
  35. Tianyi Li, Mingda Chen, Bowei Guo, and Zhiqiang Shen. 2025. A survey on diffusion language models. arXiv preprint arXiv:2508.10875. — [arXiv:2508.10875](https://arxiv.org/abs/2508.10875) find
  36. Valeria Ruscio, Umberto Nanni, and Fabrizio Silvestri. 2025. What are you sinking? a geometric approach on attention sink. Preprint, arXiv:2508.02546. — [arXiv:2508.02546](https://arxiv.org/abs/2508.02546) find
  37. Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. 2022. Diffusion-lm improves controllable text generation. Advances in neural information processing systems, 35:4328– 4343.
  38. Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. 2024. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems, 37:68658–68685.
  39. Yuerong Song, Xiaoran Liu, Ruixiao Li, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. 2025. Sparse-dllm: Accelerating diffusion llms with dynamic cache eviction. Preprint, arXiv:2508.02558. — [arXiv:2508.02558](https://arxiv.org/abs/2508.02558) find
  40. Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774.
  41. Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10–19.
  42. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint…
  43. Zeqing Wang, Gongfan Fang, Xinyin Ma, Xingyi Yang, and Xinchao Wang. 2025. Sparsed: Sparse attention for diffusion language models. arXiv preprint arXiv:2509.24014. — [arXiv:2509.24014](https://arxiv.org/abs/2509.24014) find
  44. Zining Wang, Jinyang Guo, Ruihao Gong, Yang Yong, Aishan Liu, Yushi Huang, Jiaheng Liu, and Xianglong Liu. 2024. Ptsbench: A comprehensive post-training sparsity benchmark towards algorithms and models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 5742–5751. find
  45. Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. 2025. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. Preprint, arXiv:2505.22618. — [arXiv:2505.22618](https://arxiv.org/abs/2505.22618) find
  46. Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2024a. Infllm: Training-free long-context extrapolation for llms with an efficient context memory. Advances in Neural Information Processing Systems, 37:119638–119661.
  47. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453. find
  48. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024b. Efficient streaming language models with attention sinks. Preprint, arXiv:2309.17453. find
  49. Yufei Xue, Yushi Huang, Jiawei Shao, and Jun Zhang. 2025. Vlmq: Efficient post-training quantization for large vision-language models via hessian augmentation. arXiv e-prints, pages arXiv–2508. find
  50. Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. 2025. Dream 7b: Diffusion large language models. Preprint, arXiv:2508.15487. find
  51. Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, and Chongxuan Li. 2025. Llada-v: Large language diffusion models with visual instruction tuning. arXiv preprint arXiv:2505.16933.
  52. Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. 2025. Native sparse attention: Hardware-aligned and natively trainable sparse attention. Preprint, arXiv…
  53. Jintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. 2025a. Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. In International Conference on Machine Learning (ICML). find
  54. Jintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu, and Jianfei Chen. 2025b. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations (ICLR). find
  55. Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. 2025c. Spargeattn: Accurate sparse attention accelerating any model inference. In International Conference on Machine Learning (ICML). find
  56. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, and 1 others. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:346… find
  57. Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. Preprint, arXiv:2505.19223.
  58. Lunjie Zhu, Yushi Huang, Xingtong Ge, Yufei Xue, Zhening Liu, Yumeng Zhang, Zehong Lin, and Jun Zhang. 2026. Flash-vaed: Plug-and-play vae decoders for efficient video generation. arXiv preprint arXiv:2602.19161. — [arXiv:2602.19161](https://arxiv.org/abs/2602.19161) find
  59. Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. 2025. XAttention: Block sparse attention with antidiagonal scoring. In Forty-second International Conference on Machine Learning.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/long-2026-focus-dllm-accelerating-long-context/