Elamur Research⌕
☰
home / papers / 2026 / Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement

Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement

Paper · 2026 · Qimin Zhong, Hao Liao, Haiming Qin et al.

Summary

The paper asks whether models trained by predicting tokens really build coherent internal world models. Using a linearized gradient-flow (NTK) analysis, the authors argue that Multi-Token Prediction (MTP) couples gradients across histories that share a future. This pulls their hidden states toward common belief states. Because this pressure only considers outcomes, models can also learn latent "shortcuts" that skip intermediate steps and break environment rules. The authors call this failure structural hallucination. Their remedy, LSE-MTP, adds training losses that tie each multi-step predictive representation to the teacher-forced hidden state of the real future position and to semantic embedding anchors. The extra transition layers are removed at inference. The method is tested on synthetic graph navigation and the Manhattan taxi benchmark of Vafa et al. (2024). Compared with plain MTP, it gives better latent alignment and belief compression, fewer illegal paths, and stronger resistance to injected detours. The work shows that token-level accuracy is not evidence of a consistent world model, and it offers a training-time regularizer for long-horizon state tracking.

Key points

  • Theory: under NTP, contraction mainly affects states with the same next token (Theorem 1). Under MTP, states sharing a k-step-ahead target are pulled together even when their immediate targets differ (Theorem 2, Lemma 1). The analysis assumes the lazy-training/NTK regime.
  • Architecture: a shared output head plus horizon-specific transition layers. LSE adds latent-consistency and semantic-anchoring losses, and inference uses standard autoregressive decoding.
  • Evaluation metrics on Manhattan taxi data (Table 4) include:
    • state-wise similarity;
    • Compression Precision;
    • Distinction Precision and Distinction Recall, with a probability threshold of ϵ = 0.01;
    • Detour Robustness, with random legal non-Top-1 turns injected at p = 0.01.
  • Findings: plain MTP raises similarity and compression but slightly lowers distinction precision. LSE-MTP keeps or improves distinction while compressing better and improving detour robustness.
  • Limitations stated by the authors:
    • experiments cover only graph and path-planning tasks;
    • there is no comparison with RL, contrastive, or state-space approaches;
    • the theory rests on a linear approximation of training dynamics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Microsoft Research AsiaBeijing · China82 works · 61,484 citations🎓Shenzhen UniversityShenzhen · China22 works · 95 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Qimin Zhong, Hao Liao, Haiming Qin, Mingyang Zhou, Rui Mao, Wei Chen and Naipeng Chao. 2026. Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.618

@inproceedings{zhong2026toward,
  title = {{Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement}},
  author = {Qimin Zhong, Hao Liao, Haiming Qin, Mingyang Zhou, Rui Mao, Wei Chen and Naipeng Chao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.618},
  url = {https://doi.org/10.18653/v1/2026.acl-long.618}
}

References

54 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gregor Bachmann and Vaishnavh Nagarajan. 2024. The pitfalls of next-token prediction. In Forty-first International Conference on Machine Learning. find
  2. Marc Barthelemy and Geoff Boeing. 2025. Universal model of urban street networks. Physical Review Letters, 135:137401. find
  3. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 610–623, New York, NY, USA. Association for Computing Machiner… find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  5. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple large language model inference acceleration framework with multiple decoding heads. In Forty-first International Conference on Machine Learning. find
  6. Lang Cao. 2024. GraphReason: Enhancing reasoning capabilities of large language models through a graph-based verification approach. In Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations (@ACL 2024), pages 1–12, Bangkok, Thailand. Association for Computational L… find
  7. Lénaïc Chizat, Edouard Oyallon, and Francis R. Bach. 2019. On lazy training in differentiable programming. In Conference on Neural Information Processing Systems, pages 2933–2943. find
  8. Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. 2024. Talk like a graph: Encoding graphs for large language models. In The Twelfth International Conference on Learning Representations.
  9. Arvid Frydenlund. 2025. Language models, graph searching, and supervision adulteration: When more supervision is less and how to make more more. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29011–29059, Vienna, Austria. Ass… find
  10. Prakhar Gupta, Henry Conklin, Sarah-Jane Leslie, and Andrew Lee. 2025. Better World Models Can Lead to Better Post-Training Performance. arXiv e-prints, arXiv:2512.03400. — [arXiv:2512.03400](https://arxiv.org/abs/2512.03400) find
  11. Wes Gurnee and Max Tegmark. 2024. Language models represent space and time. In The Twelfth International Conference on Learning Representations. find
  12. David Ha and Jürgen Schmidhuber. 2018. World models. CoRR, abs/1803.10122. find
  13. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. 2025. Mastering diverse control tasks through world models. Nature, 640:647–653. find
  14. Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154–8173, Singapore. Association for Computational Lingu… find
  15. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. arXiv e-prints, arXiv:1503.02531.
  16. Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. 1998. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1):99–134. find
  17. Najoung Kim and Sebastian Schuster. 2023. Entity tracking in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3835–3855, Toronto, Canada. Association for Computational Linguistics.
  18. Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun. 2024. Learning and Leveraging World Models in Visual Representation Learning. arXiv e-prints, arXiv:2403.00504. — [arXiv:2403.00504](https://arxiv.org/abs/2403.00504) find
  19. Yann LeCun and Courant. 2022. A path towards autonomous machine intelligence version 0.9.2, 2022-06-27. In Proceedings of the International Conference on Machine Intelligence. find
  20. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2:665–673. find
  21. Anson Lei, Bernhard Schölkopf, and Ingmar Posner. 2023. Variational causal dynamics: Discovering modular world models from interventions. Transactions on Machine Learning Research. find
  22. Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve. 2024. Better & faster large language models via multi-token prediction. In Forty-first International Conference on Machine Learning.
  23. Anirudh Goyal, Alex Lamb, Ying Zhang, Saizheng Zhang, Aaron C. Courville, and Yoshua Bengio. 2016. Professor forcing: A new algorithm for training recurrent networks. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc. find
  24. Belinda Z. Li, Zifan Carl Guo, and Jacob Andreas. 2025. (how) do language models track state? In Forty-second International Conference on Machine Learning. find
  25. Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pa… find
  26. Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023a. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations. find
  27. Lei Li, Jingjing Xu, Qingxiu Dong, Ce Zheng, Xu Sun, Lingpeng Kong, and Qi Liu. 2023b. Can language models understand physical concepts? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11843–11861, Singapore. Association for Computational Linguistics. find
  28. Michael L. Littman and Richard S. Sutton. 2001. Predictive representations of state. In Advances in Neural Information Processing Systems, volume 14. MIT Press. find
  29. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, and 7 others. 2022. In-con…
  30. Roma Patel and Ellie Pavlick. 2022. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations. find
  31. Jeffrey Pennington and Pratik Worah. 2017. Nonlinear random matrix theory for deep learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  32. Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2401–2410, Online. Association for Comp…
  33. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen… find
  34. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In Fourth International Conference on Learning Representations (ICLR 2016), Conference Track Proceedings, San Juan, Puerto Rico.
  35. Andrew M. Saxe, James L. McClelland, and Surya Ganguli. 2013. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv eprints, arXiv:1312.6120.
  36. Jürgen Schmidhuber. 1990. Making the world differentiable: on using self-supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments. Forschungsberichte, TU Munich, FKI 126 90:1–26. find
  37. Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, L. Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. 2019. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588:604– 609. find
  38. David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David P. Reichert, Neil C. Rabinowitz, André Barreto, and Thomas Degris. 2017. The predic-tron: End-to-end learning and planning. In Thirty-fourth International Conference on Machine Learning (I… find
  39. Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. 2023. Consistency models. In International Conference on Machine Learning, pages 32211–32252. find
  40. Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2025. On the self-verification limitations of large language models on reasoning and planning tasks. In The Thirteenth International Conference on Learning Representations. find
  41. Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S. Hu, Pratyusha Sharma, Riashat Islam, Alex Lamb, and John Langford. 2025. Next-Latent Prediction Transformers Learn Compact World Models. arXiv e-prints, arXiv:2511.05963. — [arXiv:2511.05963](https://arxiv.org/abs/2511.05963) find
  42. Keyon Vafa, Justin Y. Chen, Ashesh Rambachan, Jon Kleinberg, and Sendhil Mullainathan. 2024. Evaluating the world model implicit in a generative model. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  43. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In The Thirty-seventh Annual Conference on Neural Information Processing Systems, Datas…
  44. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation Learning with Contrastive Predictive Coding. arXiv e-prints, arXiv:1807.03748.
  45. Yilin Wen, Zifeng Wang, and Jimeng Sun. 2024. MindMap: Knowledge graph prompting sparks graph of thoughts in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10370–10388, Bangkok, Thailand. Association fo…
  46. Xixi Wu, Yifei Shen, Caihua Shan, Kaitao Song, Siwei Wang, Bohang Zhang, Jiarui Feng, Hong Cheng, Wei Chen, Yun Xiong, and Dongsheng Li. 2024. Can graph learning improve planning in large language model-based agents? In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  47. Charlie Wyatt, Aditya Joshi, and Flora Salim. 2025. Alternatives To Next Token Prediction In Text Generation – A Survey. arXiv e-prints, arXiv:2509.24435. — [arXiv:2509.24435](https://arxiv.org/abs/2509.24435) find
  48. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. 2020. On Layer Normalization in the Transformer Architecture. arXiv e-prints, arXiv:2002.04745. find
  49. Yongjing Yin, Junran Ding, Kai Song, and Yue Zhang. 2024. Semformer: Transformer language models with semantic planning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18669–18680, Miami, Florida, USA. Association for Computational Linguistics. find
  50. Guangyao Zhai, Xingyuan Zhang, and Nassir Navab. 2025. Recurrent world model with tokenized latent states. In ICLR 2025 Workshop on World Models: Understanding, Modelling and Scaling. find
  51. Fu Zhang, Jinghao Lin, and Jingwei Cheng. 2024. SALMON: A structure-aware language model with logicality and densification strategy for temporal knowledge graph reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8761–8774, Miami, Florida, USA. Association for… find
  52. Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. 2019. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4334– 4343, Florence, Italy. Association for Computational…
  53. Qimin Zhong, Hao Liao, Siwei Wang, Mingyang Zhou, Xiaoqun Wu, Rui Mao, and Wei Chen. 2025. Understanding and Enhancing the Planning Capability of Language Models via Multi-Token Prediction. arXiv e-prints, arXiv:2509.23186. — [arXiv:2509.23186](https://arxiv.org/abs/2509.23186) find
  54. Wentao Zhu, Zhining Zhang, and Yizhou Wang. 2024. Language models represent beliefs of self and others. In Forty-first International Conference on Machine Learning. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhong-2026-consistent-world-models-multi-token/