Elamur Research⌕
☰
home / papers / 2026 / MIRTH: Mutual-Information Reasoning with Temporal Hubs for Vision-Language-Action Agents

MIRTH: Mutual-Information Reasoning with Temporal Hubs for Vision-Language-Action Agents

Paper · 2026 · Hao Sun, Yu Song, Teng Shiyu et al.

Summary

Most open-source Vision-Language-Action (VLA) models choose each action from only the current frame and instruction. The authors identify three resulting problems: the agent forgets the past (for example, objects hidden by occlusion), there is no planning step between language goals and motor commands, and decoding actions token by token is slow. MIRTH extends a pretrained VLA backbone with three components:

  • Two temporal memory hubs. These compress history into fixed-length prompts. A long-term workspace hub uses exponential moving averages at several decay rates, and a short-horizon hub tracks recent motion.
  • Latent reasoning tokens. A mutual-information objective trains them to link multimodal context with action trajectories, with no text annotations required.
  • Parallel action decoding. Whole action vectors are predicted at once, replacing step-by-step scalar generation.

The authors test MIRTH on the LIBERO simulation benchmark and a real LeRobot arm. They report state-of-the-art success rates, near-perfect on LIBERO, and some ability to recover from failures. The work shows that history-aware, faster VLA control is possible without enlarging the context window.

Key points

  • Visual features come from a DINOv2+SigLIP encoder. Memory is merged into the current frame either by token prefixing or by patch infusion, and proprioception is handled in the same way.
  • In t-SNE plots, the reasoning tokens group by task meaning without task-ID supervision. For instance, opening tasks separate from pick-and-place tasks.
  • Recovery rates on LeRobot failure episodes:
    • OpenVLA: 5.2%
    • MIRTH without reasoning tokens: 8.7%
    • Full MIRTH: 12.1%
  • The authors list three limitations:
    • The latent tokens cannot be read by humans.
    • Tests cover only stationary single-arm manipulation.
    • Memory size is fixed, which limits very long tasks.
  • Code and collected datasets: github.com/kiva12138/mirth.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations

Also on the byline: Ritsumeikan University.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksEngineering · 133 worksControl and Systems Engineering · 45 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksVision & language · 657 works

Cite

Cite as Hao Sun, Yu Song, Teng Shiyu, Ziwei Niu and Yen-wei Chen. 2026. MIRTH: Mutual-Information Reasoning with Temporal Hubs for Vision-Language-Action Agents. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1016

@inproceedings{sun2026mirth,
  title = {{MIRTH: Mutual-Information Reasoning with Temporal Hubs for Vision-Language-Action Agents}},
  author = {Hao Sun, Yu Song, Teng Shiyu, Ziwei Niu and Yen-wei Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1016},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1016}
}

References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, and 1 others. 2022. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691.
  2. Katrina Ashton, Chahyon Ku, Shrey Shah, Wen Jiang, Kostas Daniilidis, and Bernadette Bucher. 2025. Helios: Hierarchical exploration for language-grounded interaction in open scenes. arXiv preprint arXiv:2509.22498. — [arXiv:2509.22498](https://arxiv.org/abs/2509.22498) find
  3. Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, and 1 others. 2022. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817. find
  4. Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, and 1 others. 2023. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In Conference on Robot Learning, pages 3909–3928. PMLR. find
  5. Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. 2025. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684– 1704. find
  6. Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, and 1 others. 2023. Palm-e: An embodied multimodal language model. find
  7. Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. 2021. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6824–6835. find
  8. Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, and 1 others. 2023. Rt-trajectory: Robotic task generalization via hindsight trajectory sketches. arXiv preprint arXiv:2311.01977. — [arXiv:2311.01977](https://arxiv.org/abs/2311.01977) find
  9. Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. 2023. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973. find
  10. Moo Jin Kim, Chelsea Finn, and Percy Liang. 2025. Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645. find
  11. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, and 1 others. 2024. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246. find
  12. Oier Mees, Dibya Ghosh, Karl Pertsch, Kevin Black, Homer Rich Walke, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, and 1 others. 2024. Octo: An open-source generalist robot policy. In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024. find
  13. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
  14. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, and 1 others. 2024. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pages 1–3…
  15. Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, and 1 others. 2024. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE International Conf… find
  16. Seungjae Lee, Yibin Wang, Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. 2024. Behavior generation with latent actions. arXiv preprint arXiv:2403.03181. — [arXiv:2403.03181](https://arxiv.org/abs/2403.03181) find
  17. Ricardo Pizarro, Roberto Valle, Rafael Barea, José M Buenaposada, Luis Baumela, and Luis M Bergasa. 2026. PO-GUISE+: Pose and object guided transformer token selection for efficient driver action recognition. IEEE Transactions on Intelligent Transportation Systems. find
  18. Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, and 1 others. Vision-language foundation models as effective robot imitators. In The Twelfth International Conference on Learning Representations. Viorica Pătrăucean, Xu Owen H… — [arXiv:2412.14294](https://arxiv.org/abs/2412.14294) find
  19. Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2022. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753. find
  20. Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. 2022. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing systems, 35:22955– 22968. find
  21. Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. 2023. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776–44791. find
  22. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  23. Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, and 1 others. 2024. Llava-plus: Learning to use tools for creating multimodal agents. In European conference on computer vision, pages 126–142. Springer. find
  24. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605. find
  25. Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. arXiv preprint arXiv:1410.3916. find
  26. Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. 2019. Long-term feature banks for detailed video understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 284–293. find
  27. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986.
  28. Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, and 1 others. 2025. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages… find
  29. Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems. Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-mirth-mutual-information-reasoning-temporal/