Elamur Research⌕
☰
home / papers / 2026 / Foresight Optimization for Strategic Reasoning in Large Language Models

Foresight Optimization for Strategic Reasoning in Large Language Models

Paper · 2026 · Jessie Wang, Jiawen Duan, Jian Wang et al.

Summary

LLMs still struggle with strategic reasoning: anticipating what another agent will do and adjusting their own choices accordingly. General reasoning-enhancement techniques do not model this forward-looking element explicitly. The authors propose Foresight Policy Optimization (FoPO), a PPO-style algorithm inspired by opponent modeling in game theory. FoPO adds a correction term that links the agent's gradient update to the update its counterpart is expected to make in response. The agent therefore optimizes for both its own payoff and its influence on the other player.

Existing game datasets such as Chess and Poker demand heavy domain expertise, so the authors built two controlled dialogue datasets, Cooperative RSA and Competitive Taboo. Training uses a shared policy in self-play: an SFT stage followed by offline RL on recorded trajectories. Experiments with Llama-3-8B-Instruct and Qwen3-14B show that FoPO beats PPO, ArCHer, GRPO and ICT both in-domain and on the out-of-domain γ-Bench. This suggests that explicitly modeling foresight yields strategic skills that transfer beyond the training games.

Key points

  • Two new datasets target distinct interaction motives.
    • Cooperative RSA is a speaker–listener object-identification game in which the pair wins by finding the target in as few turns as possible.
    • Competitive Taboo pits an attacker, who tries to make the defender say a target word, against a defender who tries to guess that word.
  • Training pipeline:
    • Role-specific prompts instantiate both agents from a single policy.
    • SFT on winning trajectories includes a KL regularizer toward the backbone model.
    • The terminal reward is propagated back to each player's earlier turns with a decay factor δ.
  • Out-of-domain evaluation: γ-Bench, using seven utility-maximizing game-theoretic tasks with five runs per task.
    • Training on both datasets combined gave the best SFT results.
    • RSA alone transferred better than Taboo alone.
  • Baseline comparison: In one block of the extracted Table 2 (the column appears to be an aggregate score), FoPO scores 64.30, compared with PPO at 62.10, ArCHer at 60.53 and ICT at 60.46.
    • PPO and ArCHer sometimes underperformed when trained with the cooperative RSA reward.
    • FoPO stayed consistently ahead.
  • Compatibility with other methods: The foresight correction also works on top of GRPO, and the resulting GR.FoPO variant outperforms plain FoPO in Taboo.
  • Code: github.com/wangjs9/ForesightOptim.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong Polytechnic UniversityHong Kong · China84 works · 4,956 citations🏢AppleCupertino · United States20 works · 2,146 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Jessie Wang, Jiawen Duan, Jian Wang, Kaitao Song, Chunpu Xu, Johnny K. W. Ho, YU Fenggang, Johan F. Hoorn and Wenjie Li. 2026. Foresight Optimization for Strategic Reasoning in Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1772

@inproceedings{wang2026foresight,
  title = {{Foresight Optimization for Strategic Reasoning in Large Language Models}},
  author = {Jessie Wang, Jiawen Duan, Jian Wang, Kaitao Song, Chunpu Xu, Johnny K. W. Ho, YU Fenggang, Johan F. Hoorn and Wenjie Li},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1772},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1772}
}

References

52 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Abdulhai and 1 others. 2024. Guess my city dataset. Accessed: 2025-09-24. find
  2. AI@Meta. 2024. Llama 3 model card. https://huggingface.co/meta-llama/Meta-Llama-3-8B. Accessed: 2025-09-23. ↗
  3. Akinator. 2007. Akinator. https://en.akinator.com. Accessed: 2025-09-24. ↗
  4. Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Jian-Guang Lou, and Dongmei Zhang. 2023. How do in-context examples affect compositional generalization? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11027– 11052. find
  5. Pengyu Cheng, Yong Dai, Tianhao Hu, Han Xu, Zhisong Zhang, Lei Han, Nan Du, and Xiaolong Li. 2024. Self-playing adversarial language game enhances llm reasoning. Advances in Neural Information Processing Systems, 37:126515–126543. find
  6. CLiPS. 2023. 20q dataset. Accessed: 2025-09-24. find
  7. Wenhao Ding, Haohong Lin, Bo Li, and Ding Zhao. 2022. Generalizing goal-conditioned reinforcement learning with variational causal reasoning. Advances in Neural Information Processing Systems, 35:26532– 26548. find
  8. Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael R. Lyu. 2025. Competing large language models in multi-agent gaming environments. In Proceedings of the Thirteenth International Conference on Learning Representations… find
  9. Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. 2024. Gtbench: Uncovering the strategic reasoning capabilities of llms via game-theoretic evaluations. Advances in Neural Information Processing Systems, 37:2… find
  10. Hyeonbin Hwang, Doyoung Kim, Seungone Kim, Seonghyeon Ye, and Minjoon Seo. 2024. Self-explore: Enhancing mathematical reasoning in language models with fine-grained rewards. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1444–1466, Miami, Florida, USA. Association fo… find
  11. Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Mguni, Yali Du, and Jun Wang. 2023. Chessgpt: Bridging policy learning and language modeling. Advances in Neural Information Processing Systems, 36:7216–7262. find
  12. Arnav Kumar Jain, Gonzalo Gonzalez-Pumariega, Wayne Chen, Alexander M Rush, Wenting Zhao, and Sanjiban Choudhury. 2025. Multi-turn code generation through single-step rewards. In Forty-second International Conference on Machine Learning. find
  13. Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. 2018. Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 122–130. find
  14. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516.
  15. Michael C. Frank and Noah D. Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998.
  16. Kanishk Gandhi, Dorsa Sadigh, and Noah Goodman. 2023. Strategic reasoning with language models. In NeurIPS 2023 Foundation Models for Decision Making Workshop. find
  17. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  18. Yuncheng Hua, Lizhen Qu, and Reza Haf. 2024. Assistive large language model agents for socially-aware negotiation dialogues. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8047–8074, Miami, Florida, USA. Association for Computational Linguistics. find
  19. Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. 2022. Adarl: What, where, and how to adapt in transfer reinforcement learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. find
  20. Chenghao Huang, Yanbo Cao, Yinlong Wen, Tao Zhou, and Yanru Zhang. 2024. Pokergpt: An end-to-end lightweight solver for multi-player texas hold’em via large language model. ArXiv preprint, abs/2401.06781. — [arXiv:2401.06781](https://arxiv.org/abs/2401.06781) find
  21. Dayeon Ki, Rachel Rudinger, Tianyi Zhou, and Marine Carpuat. 2025. Multiple LLM agents debate for equitable cultural alignment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24841–24877, Vienna, Austria. Association for Comp…
  22. Yihuai Lan, Zhiqiang Hu, Lei Wang, Yang Wang, Deheng Ye, Peilin Zhao, Ee-Peng Lim, Hui Xiong, and Hao Wang. 2024. Llm-based agent society investigation: Collaboration and confrontation in avalon gameplay. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, page… find
  23. Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. 2023. Avalonbench: Evaluating llms playing the game of avalon. ArXiv preprint, abs/2310.05036. — [arXiv:2310.05036](https://arxiv.org/abs/2310.05036) find
  24. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783.
  25. Gabriel Mukobi, Hannah Erlebach, Niklas Lauffer, Lewis Hammond, Alan Chan, and Jesse Clifton. 2023. Welfare diplomacy: Benchmarking language model cooperation. In Socially Responsible Language Modelling Research. find
  26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  27. Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. 2023. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3806–3824.
  28. Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. 2023. Avalon’s game of thoughts: Battle against deception through recursive contemplation. ArXiv preprint, abs/2310.01320. — [arXiv:2310.01320](https://arxiv.org/abs/2310.01320) find
  29. Manish Prajapat, Kamyar Azizzadenesheli, Alexander Liniger, Yisong Yue, and Anima Anandkumar. 2021. Competitive policy optimization. In Uncertainty in Artificial Intelligence, pages 64–74. PMLR. find
  30. Siyuan Wang, Enda Zhao, and Xiang Ren. 2025b. Stepwise informativeness search for improving LLM reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 25291–25309, Suzhou, China. Association for Computational Linguistics. find
  31. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1– 16. IEEE. find
  32. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. ArXiv preprint, abs/1707.06347. find
  33. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
  34. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, and 1 others. 2018. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140–1… find
  35. Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
  36. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  37. Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li. 2025a. Spa-rl: Reinforcing llm agents via stepwise progress attribution. Preprint, arXiv:2505.20732. — [arXiv:2505.20732](https://arxiv.org/abs/2505.20732) find
  38. Jiashuo Wang, Yi Cheng, and Wenjie Li. 2022. CARE: Causality reasoning for empathetic responses by conditional graph generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 729–741, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  39. Jiashuo Wang, Chunpu Xu, Chak Tou Leong, Wenjie Li, and Jing Li. 2024. Muffin: Mitigating unhelpfulness in emotional support conversations with multifaceted AI feedback. In Findings of the Association for Computational Linguistics: ACL 2024, pages 567–585, Bangkok, Thailand. Association for Computat… find
  40. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  41. Yang Xiao, Jiashuo Wang, Qiancheng Xu, Changhe Song, Chunpu Xu, Yi Cheng, Wenjie Li, and Pengfei Liu. 2025. Towards dynamic theory of mind: Evaluating LLM adaptation to temporal evolution of human states. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol…
  42. Qiancheng Xu, Yongqi Li, Heming Xia, Fan Liu, Min Yang, and Wenjie Li. 2025. PEToolLLM: Towards personalized tool learning in large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 21488–21503, Vienna, Austria. Association for Computational Linguistics. find
  43. Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. 2024. Exploring large language models for communication games: An empirical study on werewolf. Preprint, arXiv:2309.04658.
  44. Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun MA, and Bo An. 2025. Simpletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning. In NeurIPS 2025 Fourth Workshop on Deep Learning for Code. find
  45. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822. find
  46. Yuan Yao, Haoxi Zhong, Zhengyan Zhang, Xu Han, Xiaozhi Wang, Kai Zhang, Chaojun Xiao, Guoyang Zeng, Zhiyuan Liu, and Maosong Sun. 2021. Adversarial language games for advanced natural language intelligence. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conferenc… find
  47. Arianna Yuan, Will Monroe, Yue Bai, and Nate Kushman. 2018. Understanding the rational speech act model. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 40. find
  48. Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024a. Rest-mcts*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735–64772.
  49. Wenqi Zhang, Ke Tang, Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, and Weiming Lu. 2024b. Agent-pro: Learning to evolve via policy-level reflection and optimization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vo…
  50. Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Yan Xia, Wenshan Wu, Ting Song, Man Lan, and Furu Wei. 2024c. Llm as a mastermind: A survey of strategic reasoning with large language models. In First Conference on Language Modeling. find
  51. Yizhe Zhang, Jiarui Lu, and Navdeep Jaitly. 2024d. Probing the multi-turn planning capabilities of LLMs via 20 question games. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1495– 1516, Bangkok, Thailand. Association for Comp… find
  52. Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. 2024. Archer: Training language model agents via hierarchical multi-turn rl. In International Conference on Machine Learning, pages 62178–62209. PMLR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-foresight-optimization-strategic-reasoning-large/