Elamur Research⌕
☰
home / papers / 2026 / Interleaved Latent Visual Reasoning with Selective Perceptual Modeling

Interleaved Latent Visual Reasoning with Selective Perceptual Modeling

Paper · 2026 · Shuai Dong, Siyuan Wang, Xingyu Liu et al.

Summary

Multimodal LLMs can reason better when they interleave intermediate images with their textual chain of thought. However, repeatedly encoding or generating pixel-dense images is expensive. Earlier latent visual reasoning methods such as LVR and Mirage avoid pixels but have two weaknesses. They produce a latent visual signal only once, so they cannot follow evolving states. Some also squeeze a whole image into a few vectors and lose fine detail. The authors propose ILVR (Interleaved Latent Visual Reasoning), in which the model alternates text with short blocks of continuous latent embeddings that are updated at each reasoning step. Training uses ground-truth interleaved text–image trajectories, with each intermediate "helper image" replaced by latent placeholders. A momentum teacher, a smoothed copy of the model, chooses the helper-image patches most relevant to the current reasoning context. These selected patches become sparse regression targets, so no external annotation is required. ILVR outperforms prior latent and explicit-image approaches on perception and planning benchmarks, both in-domain and OOD. It also runs far faster, which suggests latent interleaving is a scalable route to multistep multimodal reasoning.

Key points

  • Latent segments are marked by <|latent_start|> and <|latent_end|>. Within a segment, the previous hidden state is fed back as the next input for a fixed K steps, and K = 8 performed best.
  • Training combines cross-entropy loss on text with an alignment loss on the latents, and the alignment weight works best at λ_sim = 1. Latent targets are chosen through top-k cosine-similarity selection of patches, guided by the momentum teacher.
  • Evaluation covers EMMA, VisuLogic and Zebra-CoT (including an OOD split). Tasks include chess puzzles, dense counting, grid navigation and robotic manipulation.
  • On a single NVIDIA H200, ILVR is roughly 8–18× faster per sample than VisionR1, PixelReasoner and Bagel-Zebra.
  • Stated limitations:
    • Experiments use only Qwen-VL backbones.
    • The method does not yet use RL optimization of latent trajectories.
    • The latents cannot be read directly by humans.
  • Code is released at github.com/XD111ds/ILVR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Shuai Dong, Siyuan Wang, Xingyu Liu, Chenglin Li, Haowen Hou and Zhongyu Wei. 2026. Interleaved Latent Visual Reasoning with Selective Perceptual Modeling. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1351

@inproceedings{dong2026interleaved,
  title = {{Interleaved Latent Visual Reasoning with Selective Perceptual Modeling}},
  author = {Shuai Dong, Siyuan Wang, Xingyu Liu, Chenglin Li, Haowen Hou and Zhongyu Wei},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1351},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1351}
}

References

43 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025a. Qwen2.5-vl technical report. ArXiv, abs/2502.13923.
  2. Sule Bai, Mingxing Li, Yong Liu, Jing Tang, Haoji Zhang, Lei Sun, Xiangxiang Chu, and Yansong Tang. 2025b. Univg-r1: Reasoning guided universal visual grounding with reinforcement learning. ArXiv, abs/2505.14231. — [arXiv:2505.14231](https://arxiv.org/abs/2505.14231) find
  3. Kedi Chen, Zhikai Lei, Xu Guo, Xuecheng Wu, Siyuan Zeng, Jia-Peng Yin, Yinqi Zhang, Qin Chen, Jie Zhou, Liang He, Qipeng Guo, Kai Chen, and Wei Zhang. 2025a. Code-driven number sequence calculation: Enhancing the inductive reasoning abilities of large language models. arXiv, abs/2510.14620. — [arXiv:2510.14620](https://arxiv.org/abs/2510.14620) find
  4. Kedi Chen, Dezhao Ruan, Yuhao Dan, Yaoting Wang, Siyu Yan, Xuecheng Wu, Yinqi Zhang, Qin Chen, Jie Zhou, Liang He, Biqing Qi, Linyang Li, Qipeng Guo, Xiaomin Shi, and Wei Zhang. 2025b. A survey of inductive reasoning for large language models. arXiv, abs/2510.10182.
  5. Liang Chen, Weichu Xie, Yiyan Liang, Hongfeng He, Han Zhao, Zhibo Yang, Zhiqi Huang, Haoning Wu, Haoyu Lu, Y. Charles, Yiping Bao, Yu Fan, Guopeng Li, Haiyang Shen, Xuanzhong Chen, Wen Xu, Shuzheng Si, Zefan Cai, Wenhao Chai, and 10 others. 2026. Babyvision: Visual reasoning beyond language. arXiv,… — [arXiv:2601.06521](https://arxiv.org/abs/2601.06521) find
  6. Jeffrey Cheng and Benjamin Van Durme. 2024. Compressed chain of thought: Efficient reasoning through dense representations. ArXiv, abs/2412.13171. find
  7. Zihui Cheng, Qiguang Chen, Jin Zhang, Hao Fei, Xiaocheng Feng, Wanxiang Che, Min Li, and Libo Qin. 2024. Comt: A novel benchmark for chain of multi-modal thought on large vision-language models. ArXiv, abs/2412.12932. — [arXiv:2412.12932](https://arxiv.org/abs/2412.12932) find
  8. Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. 2024. Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation. ArXiv, abs/2407.06135. — [arXiv:2407.06135](https://arxiv.org/abs/2407.06135) find
  9. Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Shi Guang, and Haoqi Fan. 2025. Emerging properties in unified multimodal pretraining. ArXiv, abs/2505.14683. — [arXiv:2505.14683](https://arxiv.org/abs/2505.14683) find
  10. Sicheng Feng, Kaiwen Tuo, Song Wang, Lingdong Kong, Jianke Zhu, and Huan Wang. 2025. Re-wardmap: Tackling sparse rewards in fine-grained visual reasoning via multi-stage reinforcement learning. ArXiv, abs/2510.02240. — [arXiv:2510.02240](https://arxiv.org/abs/2510.02240) find
  11. Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Cor-ring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinei A. F. Florêncio, and Cha Zhang. 2025. Refocus: Visual editing as a chain of thought for structured image understanding. ArXiv, abs/2501.05452. find
  12. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason E. Weston, and Yuandong Tian. 2024. Training large language models to reason in a continuous latent space. ArXiv, abs/2412.06769. find
  13. Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. 2025. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. ArXiv, abs/2501.05444. — [arXiv:2501.05444](https://arxiv.org/abs/2501.05444) find
  14. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2019. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726–9735.
  15. Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke S. Zettlemoyer, Noah A. Smith, and Ranjay Krishna. 2024. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. ArXiv, abs/2406.09403. find
  16. Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaoshen Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. 2025a. Vision-r1: Incentivizing reasoning capability in multimodal large language models. ArXiv, abs/2503.06749. find
  17. Zeyi Huang, Yuyang Ji, Anirudh Sundara Rajan, Zefan Cai, Wen Xiao, Junjie Hu, and Yong Jae Lee. 2025b. Visualtoolagent (vista): A reinforcement learning framework for visual tool selection. ArXiv, abs/2505.20289. — [arXiv:2505.20289](https://arxiv.org/abs/2505.20289) find
  18. Ang Li, Charles L. Wang, Kaiyu Yue, Zikui Cai, Ollie Liu, Deqing Fu, Peng Guo, Wang Bill Zhu, Vatsal Sharan, Robin Jia, Willie Neiswanger, Furong Huang, Tom Goldstein, and Micah Goldblum. 2025a. Zebra-cot: A dataset for interleaved vision language reasoning. ArXiv, abs/2507.16746. — [arXiv:2507.16746](https://arxiv.org/abs/2507.16746) find
  19. Bangzheng Li, Ximeng Sun, Jiang Liu, Ze Wang, Jialian Wu, Xiaodong Yu, Hao Chen, Emad Barsoum, Muhao Chen, and Zicheng Liu. 2025b. Latent visual reasoning. ArXiv, abs/2509.24251. — [arXiv:2509.24251](https://arxiv.org/abs/2509.24251) find
  20. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024. Llava-onevision: Easy visual task transfer. ArXiv, abs/2408.03326.
  21. Chenglin Li, Feng Han, Feng Tao, Ruilin Li, Qianglong Chen, Jingqi Tong, Yin Zhang, and Jiaqi Wang. 2025c. Videopro: Adaptive program reasoning for long video understanding.
  22. Dairu Liu, Ziyue Wang, Minyuan Ruan, Fuwen Luo, Chi Chen, Peng Li, and Yang Liu. 2025. Visual abstract thinking empowers multimodal reasoning. ArXiv, abs/2505.20164. — [arXiv:2505.20164](https://arxiv.org/abs/2505.20164) find
  23. Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. 2024a. Visual cot: Advancing multimodal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. Advances in Neural Information Processing Systems 37. find
  24. Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. 2024b. Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models. ArXiv, abs/2403.16999. find
  25. Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. 2025. Codi: Compressing chain-of-thought into continuous space via self-distillation. ArXiv, abs/2502.21074.
  26. Alex Su, Haozhe Wang, Weiming Ren, Fangzhen Lin, and Wenhu Chen. 2025. Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning. ArXiv, abs/2505.15966. — [arXiv:2505.15966](https://arxiv.org/abs/2505.15966) find
  27. Dawei Wang, Chengmin Zhou, Ding Zhao, Xinyu Liu, Marci Chi Ma, Gary Ushaw, and Rich Davison. 2026a. Towermind: A tower defence game learning environment and benchmark for llm as agents. In AAAI Conference on Artificial Intelligence. find
  28. Dianyi Wang, Wei Song, Yikun Wang, Siyuan Wang, Kaicheng yu, Zhongyu Wei, and Jiaqi Wang. 2025a. Autoregressive semantic visual reconstruction helps vlms understand better. arXiv, abs/2506.09040.
  29. Jiacong Wang, Zijiang Kang, Haochen Wang, Haiyong Jiang, Jiawen Li, Bohong Wu, Ya Wang, Jiao Ran, Xiao Liang, Chao Feng, and Jun Xiao. 2025b. Vgr: Visual grounded reasoning. ArXiv, abs/2506.11991. — [arXiv:2506.11991](https://arxiv.org/abs/2506.11991) find
  30. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, and 44 others. 2025c. Internvl3.5: Advancing open-source multimodal…
  31. Yubo Wang, Juntian Zhang, Yichen Wu, Yankai Lin, Nils Lukas, and Yuhan Liu. 2026b. Forest before trees: Latent superposition for efficient visual reasoning. arXiv, abs/2601.06803.
  32. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903.
  33. Qiucheng Wu, Handong Zhao, Michael Stephen Saxon, Trung M. Bui, William Yang Wang, Yang Zhang, and Shiyu Chang. 2024. Vsp: Assessing the dual challenges of perception and reasoning in spatial planning tasks for vlms. ArXiv, abs/2407.01863. — [arXiv:2407.01863](https://arxiv.org/abs/2407.01863) find
  34. Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wen gang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, Wenhai Wang, Jifeng Dai, and Jinguo Zhu. 2025. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. ArXiv, abs/2504.15279. — [arXiv:2504.15279](https://arxiv.org/abs/2504.15279) find
  35. Yuqi Xu, Rizhen Hu, Zihan Liu, Mou Sun, and Kun Yuan. 2026. Grouter: Decoupling routing from representation for accelerated moe training. arXiv preprint arXiv:2603.06626. — [arXiv:2603.06626](https://arxiv.org/abs/2603.06626) find
  36. Zeyuan Yang, Xueyang Yu, Delin Chen, Maohao Shen, and Chuang Gan. 2025. Machine mental imagery: Empower multimodal reasoning with latent visual tokens. ArXiv, abs/2506.17218. — [arXiv:2506.17218](https://arxiv.org/abs/2506.17218) find
  37. Guanghao Zhang, Tao Zhong, Yan Xia, Zhelun Yu, Haoyuan Li, Wanggui He, Fangxun Shu, Mushui Liu, Dong She, Yi Wang, and Hao Jiang. 2025a. Cmmcot: Enhancing complex multi-image comprehension via multi-modal chain-of-thought and memory augmentation. ArXiv, abs/2503.05255. — [arXiv:2503.05255](https://arxiv.org/abs/2503.05255) find
  38. Huanyu Zhang, Wenshan Wu, Chengzu Li, Ning Shang, Yan Xia, Yangyu Huang, Yifan Zhang, Li Dong, Zhang Zhang, Liang Wang, Tien-Ping Tan, and Furu Wei. 2025b. Latent sketchpad: Sketching visual thoughts to elicit multimodal reasoning in mllms. ArXiv, abs/2510.24514. — [arXiv:2510.24514](https://arxiv.org/abs/2510.24514) find
  39. Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alexander J. Smola. 2023. Multimodal chain-of-thought reasoning in language models.
  40. Trans. Mach. Learn. Res., 2024. find
  41. Di Zhao, Jingfeng Zhang, Hongsheng Hu, Philippe Fournier-Viger, Gill Dobbie, and Yun Sing Koh. 2025. Balancing invariant and specific knowledge for domain generalization with online knowledge distillation. In International Joint Conference on Artificial Intelligence. find
  42. Yakun Zhu, Yutong Huang, Shengqian Qin, Zhongzhen Huang, Shaoting Zhang, and Xiaofan Zhang. 2026. Medmcp-calc: Benchmarking llms for realistic medical calculator scenarios via mcp integration. arXiv, abs/2601.23049.
  43. Yakun Zhu, Zhongzhen Huang, Qianhan Feng, Linjie Mu, Yannian Gu, Shaoting Zhang, Qi Dou, and Xiaofan Zhang. 2025. Cp-env: Evaluating large language models on clinical pathways in a controllable hospital environment. arXiv, abs/2512.10206. — [arXiv:2512.10206](https://arxiv.org/abs/2512.10206) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dong-2026-interleaved-latent-visual-reasoning-selective/