Elamur Research⌕
☰
home / papers / 2024 / Mastering Board Games by External and Internal Planning with Language Models

Mastering Board Games by External and Internal Planning with Language Models

Paper · 2024 · John Schultz, Jakub Adamek, Matej Jusup et al.

Summary

The paper asks whether a language model can plan well enough to play board games at a high level without help from a game engine during play. The authors pre-train a Transformer called MAV (multi-action-value model) only on textual game records for Chess, Chess960, Connect Four and Hex. Training targets come from engine annotations. A single MAV call can update the board after a move, list legal moves, detect terminal positions, score the top-k candidate moves with discretized win probabilities, and name the best action.

MAV is used for search in two ways. In external search, it serves as the transition and value function inside an asynchronous MCTS loop. In internal search, it learns to write out a linearized minimax tree in context and then commit to a move, extending the Stream of Search idea to much harder games.

Both approaches clearly strengthen play over the base model. With MCTS, chess play reaches Grandmaster level using a search budget close to that of human players. The authors suggest that pairing search with learned domain knowledge could also help LLM reasoning outside games.

Key points

  • Action values are classified into 64 bucket tokens rather than regressed. For chess, Stockfish centipawn scores are converted to win probabilities using the Lichess formula. Connect Four is labelled with Fhourstones and Hex with neurobenzene.
  • Two design choices address specific problems:
    • Move order in training examples is randomized to reduce hallucinations.
    • A %best_action command teaches the model to convert winning positions without engine tie-breaking.
  • Mean scoring (the expectation over buckets) is compared with max scoring (the mode). Mean scoring separates moves better in near-equal positions.
  • MAV–MCTS considers roughly 100–1k moves per decision, compared with about 10k simulations for AlphaZero. It gains around +300 Elo even with this limited budget, and strength keeps rising as simulations increase in every game.
  • Internal search (MAV–IS) was tested at breadth up to 4 and depth 2, and its Elo grows with the token budget. It produced a correctly shaped search tree in 99.6% of 5,980 sampled positions. On chess world-modeling metrics it is about 99.5–100% accurate, including on Lichess puzzles and out-of-distribution random boards.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States533 works · 443,885 citations🏢Google DeepMindLondon · United Kingdom82 works · 84,829 citations🎓ETH ZurichZurich · Switzerland97 works · 9,612 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as John Schultz, Jakub Adamek, Matej Jusup, Marc Lanctot, Michael Kaisers, Sarah Perrin, Daniel Hennes, Jeremy Shar, Cannada Lewis, Anian Ruoss, Tom Zahavy, Petar Veličkovic, Laurel Prince, Satinder Singh, Eric Malmi and Nenad Tomašev. 2024. Mastering Board Games by External and Internal Planning with Language Models. arXiv:2412.12119.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.12119 · arXiv 2412.12119

@misc{schultz2024mastering,
  title = {{Mastering Board Games by External and Internal Planning with Language Models}},
  author = {John Schultz, Jakub Adamek, Matej Jusup, Marc Lanctot, Michael Kaisers, Sarah Perrin, Daniel Hennes, Jeremy Shar, Cannada Lewis, Anian Ruoss, Tom Zahavy, Petar Velivckovic, Laurel Prince, Satinder Singh, Eric Malmi and Nenad Tomavsev},
  year = {2024},
  doi = {10.48550/arxiv.2412.12119},
  eprint = {2412.12119},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.12119}
}

References

112 references, 48 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021. — [arXiv:2104.13478](https://arxiv.org/abs/2104.13478) find
  2. Top chess engine championship. URL https://tcec-chess.com/ . Torres and his remarkable automatic devices — he would substitute machinery for the human mind. Scientific American Supplement, 80 (2079):296, November 1915. ↗
  3. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  4. M. Acher. Debunking the chessboard: Confronting gpts against chess engines to estimate elo ratings and assess legal move abilities, C. Burns, P. Izmailov, J. H. Kirchner, B. Baker, 2023. URL — Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision https://blog.mathieuacher. L. Gao, L. Aschenbrenner, Y. Chen, A. Ecof-com/GPTsChessEloRatingLegalMoves/ . fet… ↗
  5. Lin, A. Lv, Y. Wu, 2023. URL https://arxiv.org/abs/2302. X. Gao, J.-R. Wen, R. Yan, and Y. Li. Masked 04023 . thought: Simply masking partial reasoning steps can improve mathematical reasoning M. Besta, N. Blach, A. Kubicek, R. Gersten-learning of language models. arXiv preprint berger, M. Podstawsk… — [arXiv:2403.02178](https://arxiv.org/abs/2403.02178) ↗
  6. AAAI Press, 2024. ning of large language model on knowledge graphs, 2024b. URL https://arxiv.org/ abs/2410.23875 . — [arXiv:2410.23875](https://arxiv.org/abs/2410.23875) ↗
  7. Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu. Self-play fine-tuning converts weak language models to strong language models, 2024c. URL https://arxiv.org/abs/2401.01335 . ↗
  8. Z. Chen, M. White, R. Mooney, A. Payani, Y. Su, and H. Sun. When is tree search useful for llm planning? it depends on the discriminator, 2024d. URL https://arxiv.org/abs/ 2402.10890 . ↗
  9. A. E. Elo. The proposed uscf rating system, its development, theory, and applications. Chess life, 22(8):242–247, 1967. find
  10. N. Ensmenger. Is chess the drosophila of artificial intelligence? a social history of an algorithm. Social studies of science, 42(1):5–30, 2012. find
  11. M. Fang, S. Deng, Y. Zhang, Z. Shi, L. Chen, M. Pechenizkiy, and J. Wang. Large language models are neurosymbolic reasoners. In AAAI, pages 17985–17993. AAAI Press, 2024.
  12. Y. K. Chia, V. T. Y. Han, D. Ghosal, L. Bing, and S. Poria. Puzzlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns, 2024. URL https://arxiv.org/abs/2403.13315 . ↗
  13. J. Farebrother, J. Orbay, Q. Vuong, A. A. Taïga, Y. Chebotar, T. Xiao, A. Irpan, S. Levine, P. S. Castro, A. Faust, A. Kumar, and R. Agarwal. Stop regressing: Training value functions via classification for scalable deep rl, 2024. URL https://arxiv.org/abs/2403.03950 . — [arXiv:2403.03950](https://arxiv.org/abs/2403.03950) ↗
  14. A. Costarelli, M. Allen, R. Hauksson, G. Sodunke, S. Hariharan, C. Cheng, W. Li, and A. Yadav. Gamebench: Evaluating strategic reasoning abilities of LLM agents. arXiv preprint arXiv:2406.06613, 2024.
  15. G. Feng, B. Zhang, Y. Gu, H. Ye, D. He, and L. Wang. Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems, 36, 2024a. find
  16. W. Cui, K. Bi, J. Guo, and X. Cheng. More: Multi-modal retrieval augmented generative commonsense reasoning. arXiv preprint arXiv:2402.13625, 2024. find
  17. X. Feng, Y. Luo, Z. Wang, H. Tang, M. Yang, K. Shao, D. Mguni, Y. Du, and J. Wang. Chessgpt: Bridging policy learning and language modeling, 2023. URL https://arxiv.org/ abs/2306.09200 . ↗
  18. J. Czech, J. Blüml, K. Kersting, and H. Stein-grimsson. Representation matters for mastering chess: Improved feature representation in alphazero outperforms switching to transformers. arXiv preprint arXiv:2304.14918, 2024. — [arXiv:2304.14918](https://arxiv.org/abs/2304.14918) find
  19. G. Dagan, F. Keller, and A. Lascarides. Dynamic planning with a llm, 2023. URL https://arxiv.org/abs/2308.06391 . ↗
  20. R. Ding, C. Zhang, L. Wang, Y. Xu, M. Ma, W. Zhang, S. Qin, S. Rajmohan, Q. Lin, and D. Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint arXiv:2311.04254, 2023. find
  21. J. Duan, R. Zhang, J. Diffenderfer, B. Kailkhura, L. Sun, E. Stengel-Eskin, M. Bansal, T. Chen, and K. Xu. Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations, 2024. URL https://arxiv.org/abs/2402.12348 . ↗
  22. X. Feng, Z. Wan, M. Wen, S. M. McAleer, Y. Wen, W. Zhang, and J. Wang. Alphazero-like tree-search can guide large language model decoding and training, 2024b. URL https://arxiv.org/abs/2309.17179 . ↗
  23. E. Gan, Y. Zhao, L. Cheng, Y. Mao, A. Goyal, K. Kawaguchi, M.-Y. Kan, and M. Shieh. Reasoning robustness of llms to adversarial typographical errors, 2024. URL https://arxiv.org/abs/2411.05345 . ↗
  24. K. Gandhi, D. Sadigh, and N. D. Goodman. Strategic reasoning with language models. arXiv preprint arXiv:2305.19165, 2023.
  25. K. Gandhi, J.-P. Fränken, T. Gerstenberg, and N. Goodman. Understanding social reasoning in language models with language models. Advances in Neural Information Processing Systems, 36, 2024a. find
  26. K. Gandhi, D. Lee, G. Grand, M. Liu, W. Cheng, A. Sharma, and N. D. Goodman. Stream of search (SoS): Learning to search in language, 2024b.
  27. C. Gao and J. Pawlewicz. neurobenzene: An improved benzene project for playing and solving hex with the help of deep neural networks. URL — Making Pre-trained Language Models Better Few-shot Learners https://github.com/cgao3/ neurobenzene . T. Gao, A. Fisch, and D. Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.157… ↗
  28. Gemini Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: A family of highly capable multimodal models, 2024. URL https://arxiv.org/abs/2312.11805 . et al. A survey on large language models: Applications, challenges, limitations, a… ↗
  29. Authorea Preprints, 2023. find
  30. K. Hamade, R. McIlroy-Young, S. Sen, J. Kleinberg, and A. Anderson. Designing skill-compatible ai: Methodologies and frameworks in chess, 2024. URL https://arxiv.org/abs/2405. 05066 . ↗
  31. S. Hao, Y. Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023. find
  32. S. Haresh, D. Dijkman, A. Bhattacharyya, and R. Memisevic. Clevrskills: Compositional language and visual reasoning in robotics. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=64sZtFSOh6 . ↗
  33. P. Giadikiaroglou, M. Lymperaiou, G. Filandrianos, and G. Stamou. Puzzle solving using reasoning of large language models: A survey, J. J. Horton. Large language models as simu-2024. URL https://arxiv.org/abs/2402. lated economic agents: What can we learn from 11291 . homo silicus? Technical report,… ↗
  34. M. U. Hadi, R. Qureshi, A. Shah, M. Irfan, A. Zafar, M. B. Shaikh, N. Akhtar, J. Wu, S. Mirjalili, S. Hu, T. Huang, and L. Liu. Pokellmon: A human-parity agent for pokemon battles with large language models, 2024c. URL https://arxiv.org/abs/2402.01118 . reasoners. Advances in neural information pro… — [arXiv:2402.01118](https://arxiv.org/abs/2402.01118) ↗
  35. J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou. Large language models cannot self-correct reasoning yet, 2024. URL https://arxiv.org/abs/2310.01798 . ↗
  36. A. Kumar, V. Zhuang, R. Agarwal, Y. Su, J. D. Co-Reyes, A. Singh, K. Baumli, S. Iqbal, C. Bishop, R. Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917, 2024.
  37. E. Jenner, S. Kapur, V. Georgiev, C. Allen, S. Emmons, and S. Russell. Evidence of learned M. Lanctot, E. Lockhart, J.-B. Lespiau, V. Zam-look-ahead in a chess-playing neural network, baldi, S. Upadhyay, J. Pérolat, S. Srini-2024. URL https://arxiv.org/abs/2406. vasan, F. Timbers, K. Tuyls, S. Omids… — [arXiv:1908.09453](https://arxiv.org/abs/1908.09453) ↗
  38. Z. Jin, Y. Chen, F. Leeb, L. Gresele, O. Kamal, Z. Lyu, K. Blin, F. Gonzalez Adauto, M. Kleiman-Weiner, M. Sachan, et al. Cladder: A benchmark to assess causal reasoning capabilities of language models. Advances in Neural Information Processing Systems, 36, 2024. find
  39. D. Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011. find
  40. V. Mnih. In-context reinforcement learning with algorithm distillation, 2022. URL https://arxiv.org/abs/2210.14215 . — [arXiv:2210.14215](https://arxiv.org/abs/2210.14215) ↗
  41. J. Lee, F. Yang, T. Tran, Q. Hu, E. Barut, K.-W. Chang, and C. Su. Can small language models help large language models reason better?: Lm-guided chain-of-thought. arXiv preprint arXiv:2404.03414, 2024. find
  42. L. Lehnert, S. Sukhbaatar, D. Su, Q. Zheng, P. Mcvay, M. Rabbat, and Y. Tian. Beyond a ∗ : Better planning with transformers via search dynamics bootstrapping, 2024. URL https://arxiv.org/abs/2402.14083 . ↗
  43. J. Li, H. Le, Y. Zhou, C. Xiong, S. Savarese, and D. Sahoo. Codetree: Agent-guided tree search for code generation with large language models, 2024a. URL https://arxiv.org/abs/ 2411.04329 . ↗
  44. J. Kil, Z. Mai, J. Lee, A. Chowdhury, Z. Wang, K. Cheng, L. Wang, Y. Liu, and W.-L. Chao. MLLM-compbench: A comparative reasoning benchmark for multimodal LLMs. In The Thirty-eight Conference on Neural Information N. Li, C. Gao, Y. Li, and Q. Liao. Large lan-Processing Systems Datasets and Benchmark… — Tree Search for Language Model Agents
  45. J. Light, M. Cai, W. Chen, G. Wang, X. Chen, W. Cheng, Y. Yue, and Z. Hu. Strategist: Learning strategic skills by llms via bi-level tree search, 2024. URL https://arxiv.org/ abs/2408.10635 . — [arXiv:2408.10635](https://arxiv.org/abs/2408.10635) ↗
  46. J. Long. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.08291, 2023. find
  47. P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022. find
  48. P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.-C. Zhu, and J. Gao. Chameleon: Plug-and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36, 2024.
  49. L. Markeeva, S. McLeish, B. Ibarz, W. Bounsi, O. Kozlova, A. Vitvitskyi, C. Blundell, T. Goldstein, A. Schwarzschild, and P. Veličković. The clrs-text algorithmic reasoning language benchmark. arXiv preprint arXiv:2406.04229, 2024. find
  50. S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024.
  51. S. A. Mirsoleimani, A. Plaat, J. van den Herik, and J. Vermaseren. An analysis of virtual loss in parallel MCTS. In International Conference on Agents and Artificial Intelligence, 2017. URL https://api.semanticscholar.org/ CorpusID:34042014 . ↗
  52. C. Mitra, B. Huang, T. Darrell, and R. Herzig. Compositional chain-of-thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420–14431, 2024. find
  53. D. Monroe and T. Leela Chess Zero Team. Mastering chess with a transformer model, 2024. URL https://arxiv.org/abs/2409.12272 . — [arXiv:2409.12272](https://arxiv.org/abs/2409.12272) ↗
  54. M. Nguyen and E. Shareghi. One step at a time: Language agents are stepwise planners, 2024. URL https://arxiv.org/abs/2411. 08432 . ↗
  55. H. Nori, N. Usuyama, N. King, S. M. McKinney, X. Fernandes, S. Zhang, and E. Horvitz. From medprompt to o1: Exploration of run-time strategies for medical challenge problems and beyond, 2024. URL https://arxiv.org/ abs/2411.03590 . — [arXiv:2411.03590](https://arxiv.org/abs/2411.03590) ↗
  56. E. Markowitz, A. Ramakrishna, J. Dhamala, N. Mehrabi, C. Peris, R. Gupta, K.-W. Chang, and A. Galstyan. Tree-of-traversals: A zero-shot reasoning algorithm for augmenting black- B. Paranjape, S. Lundberg, S. Singh, H. Hajishirzi, L. Zettlemoyer, and M. T. Ribeiro. Art: box language models with knowl… — ART: Automatic multi-step reasoning and tool-use for large language models
  57. S. Menon, R. Zemel, and C. Vondrick. Whiteboard-of-thought: Thinking step-by-step across modalities, 2024. URL https://arxiv.org/ abs/2406.14562 . ↗
  58. J. Pfau, W. Merrill, and S. R. Bowman. Let’s think dot by dot: Hidden computation in transformer language models, 2024. URL https://arxiv.org/abs/2404.15758 . ↗
  59. A. Plaat, A. Wong, S. Verberne, J. Broekens, N. van Stein, and T. Back. Reasoning with large language models, a survey. arXiv preprint arXiv:2407.11511, 2024.
  60. O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022. find
  61. B. Prystawski, M. Li, and N. Goodman. Why think step by step? reasoning emerges from the locality of experience. Advances in Neural Information Processing Systems, 36, 2024.
  62. P. Putta, E. Mills, N. Garg, S. Motwani, C. Finn, D. Garg, and R. Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents, 2024. URL https://arxiv.org/ abs/2408.07199 . ↗
  63. R. Rafailov, J. Hejna, R. Park, and C. Finn. From 𝑟 to 𝑞 ∗ : Your language model is secretly a q-function, 2024. URL — From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function https://arxiv.org/ abs/2404.12358 . ↗
  64. K. Saab, T. Tu, W.-H. Weng, R. Tanno, D. Stutz, E. Wulczyn, F. Zhang, T. Strother, C. Park, E. Vedadi, et al. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416, 2024.
  65. A. Saparov and H. He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. arXiv preprint arXiv:2210.01240, 2022.
  66. T. Schaul, J. Togelius, and J. Schmidhuber. Measuring intelligence through games. arXiv preprint arXiv:1109.1314, 2011. — [arXiv:1109.1314](https://arxiv.org/abs/1109.1314) find
  67. J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020. doi: 10.1038/ s41586-020-03051-4. URL htt… find
  68. H. Shao, S. Qian, H. Xiao, G. Song, Z. Zong, S. Rasal and E. J. Hauer. Optimal decision making L. Wang, Y. Liu, and H. Li. Visual cot: Ad-through scenario simulations using large lan-vancing multi-modal language models with guage models, 2024. URL https://arxiv.a comprehensive dataset and benchmark… — [arXiv:2407.06486](https://arxiv.org/abs/2407.06486) ↗
  69. A. Ruoss, G. Delétang, S. Medapati, J. Grau-Moya, L. K. Wenliang, E. Catt, J. Reid, C. A. Lewis, J. Veness, and T. Genewein. Amortized planning with large-scale transformers: A case study on chess. In NeurIPS, 2024a. find
  70. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024b.
  71. A. Ruoss, F. Pardo, H. Chan, B. Li, V. Mnih, and T. Genewein. LMAct: A benchmark for in-context imitation learning with long multimodal demonstrations, 2024b. URL https://arxiv.org/abs/2412.01441 . — [arXiv:2412.01441](https://arxiv.org/abs/2412.01441) ↗
  72. F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, et al. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057, 2022.
  73. W. Shi, Z. Hu, Y. Bin, J. Liu, Y. Yang, S.-K. Ng, L. Bing, and R. K.-W. Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models, 2024. URL https://arxiv.org/abs/2406.17294 . ↗
  74. O. Topsakal and J. B. Harper. Benchmarking large language model (llm) performance for game playing via tic-tac-toe. Electronics, 13(8):1532, 2024. find
  75. J. Tromp. The fhourstones benchmark. URL D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017. URL…
  76. A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, X. Garcia, P. J. Liu, J. Harrison, J. Lee, K. Xu, A. Parisi, A. Kumar, A. Alemi, A. Rizkowsky, A. Nova, B. Adlam, B. Bohnet, G. Elsayed, H. Sedghi, I. Mordatch, I. Simpson, I. Gur, J. Snoek, J. Pennington, J. Hron, K. Kenealy, K. Swersky, K.… — Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
  77. K. Stechly, K. Valmeekam, and S. Kambhampati. Chain of thoughtlessness? an analysis of cot in planning, 2024. URL https://arxiv.org/ abs/2405.04776 . — [arXiv:2405.04776](https://arxiv.org/abs/2405.04776) ↗
  78. X. Tang, X. Wang, W. X. Zhao, and J.-R. Wen. Dawn-icl: Strategic planning of problem-solving trajectories for zero-shot in-context learning, 2024a. URL https://arxiv.org/ abs/2410.20215 . ↗
  79. Z. Tang, X. Zhang, B. Wan, and F. Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.02884, 2024b. https://tromp.github.io/c4/fhour. html . ↗
  80. C. F. Tsai, X. Zhou, S. S. Liu, J. Li, M. Yu, and H. Mei. Can large language models play text games well? current state-of-the-art and open questions. arXiv preprint arXiv:2304.02868, 2023a. — [arXiv:2304.02868](https://arxiv.org/abs/2304.02868) find
  81. C. F. Tsai, X. Zhou, S. S. Liu, J. Li, M. Yu, and H. Mei. Can large language models play text games well? current state-of-the-art and open questions, 2023b. URL https://arxiv.org/abs/2304.02868 . — [arXiv:2304.02868](https://arxiv.org/abs/2304.02868) ↗
  82. M. Turpin, J. Michael, E. Perez, and S. Bowman. Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36, 2024. — Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
  83. P. Veličković, C. Perivolaropoulos, F. Barbero, and R. Pascanu. softmax is not enough (for sharp out-of-distribution). arXiv preprint arXiv:2410.01104, 2024. — [arXiv:2410.01104](https://arxiv.org/abs/2410.01104) find
  84. D. Wang, Z. Ye, F. Fang, and L. Li. Cooperative strategic planning enhances reasoning capabilities in large language models, 2024a. URL https://arxiv.org/abs/2410.20007 . — [arXiv:2410.20007](https://arxiv.org/abs/2410.20007) ↗
  85. H. Wang, S. Feng, T. He, Z. Tan, X. Han, and Y. Tsvetkov. Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems, 36, 2024b.
  86. H. Wang, J. Xu, S. Xie, R. Wang, J. Li, Z. Xie, B. Zhang, C. Xiong, and X. Chen. M4u: Evaluating multilingual understanding and reasoning for large multimodal models, 2024c. URL https://arxiv.org/abs/2405.15638 . — [arXiv:2405.15638](https://arxiv.org/abs/2405.15638) ↗
  87. Y. Tian, B. Peng, L. Song, L. Jin, D. Yu, H. Mi, and D. Yu. Toward self-improvement of llms K. Wang, J. Pan, W. Shi, Z. Lu, M. Zhan, and H. Li. via imagination, searching, and criticizing, Measuring multimodal mathematical reason-2024. URL https://arxiv.org/abs/2404. ing with math-vision dataset. ar… ↗
  88. L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.-W. Lee, and E.-P. Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXiv:2305.04091, 2023. find
  89. Y. Xie, A. Goyal, W. Zheng, M.-Y. Kan, T. P. Lillicrap, K. Kawaguchi, and M. Shieh. Monte carlo tree search boosts reasoning via iterative preference learning, 2024b. URL https://arxiv.org/abs/2405.00451 . ↗
  90. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
  91. S. Xiong, A. Payani, R. Kompella, and F. Fekri. Large language models can learn temporal reasoning. arXiv preprint arXiv:2401.06853, 2024. find
  92. Y. Wang, S. Zhao, Z. Wang, H. Huang, M. Fan, Y. Zhang, Z. Wang, H. Wang, and T. Liu. Strategic chain-of-thought: Guiding accurate reasoning in llms through strategy elicitation, 2024e. URL https://arxiv.org/ abs/2409.03271 . — [arXiv:2409.03271](https://arxiv.org/abs/2409.03271) ↗
  93. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.
  94. S. Wu, L. Zhu, T. Yang, S. Xu, Q. Fu, Y. Wei, and H. Fu. Enhance reasoning for large language models in the game werewolf, 2024a. URL https://arxiv.org/abs/2402.02330 . — [arXiv:2402.02330](https://arxiv.org/abs/2402.02330) ↗
  95. X. Wu, Y. Shen, C. Shan, K. Song, S. Wang, B. Zhang, J. Feng, H. Cheng, W. Chen, Y. Xiong, and D. Li. Can graph learning improve planning in LLM-based agents? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https://openreview.net/forum?id= bmoS6Ggw4j . ↗
  96. Y. Wu, J. Liu, X. Bu, J. Liu, Z. Zhou, Y. Zhang, C. Zhang, Z. Bai, H. Chen, T. Ge, et al. Con-ceptmath: A bilingual concept-wise benchmark for measuring mathematical reasoning of large language models. arXiv preprint arXiv:2402.14660, 2024c. — [arXiv:2402.14660](https://arxiv.org/abs/2402.14660) find
  97. C. Yang, X. Wang, J. Jiang, Q. Zhang, and X. Huang. Evaluating world models with llm for decision making, 2024a. URL https://arxiv.org/abs/2411.08794 . — [arXiv:2411.08794](https://arxiv.org/abs/2411.08794) ↗
  98. L. Yang, Z. Yu, T. Zhang, S. Cao, M. Xu, W. Zhang, J. E. Gonzalez, and B. Cui. Buffer of thoughts: Thought-augmented reasoning with large language models, 2024b. URL https://arxiv.org/abs/2406.04271 . ↗
  99. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
  100. S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. find
  101. X. Ye and G. Durrett. The unreliability of explanations in few-shot prompting for textual reasoning. Advances in neural information processing systems, 35:30378–30392, 2022.
  102. E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022. URL https://arxiv.org/abs/2203. 14465 . ↗
  103. E. Zelikman, G. Harik, Y. Shao, V. Jayasiri, N. Haber, and N. D. Goodman. Quiet-star: Language models can teach themselves to think before speaking, 2024. URL https://arxiv.org/abs/2403.09629 . ↗
  104. J. Xie, R. Zhang, Z. Chen, X. Wan, and G. Li. Whodunitbench: Evaluating large multimodal agents via murder mystery games. In The Z. Zeng, Y. Liu, Y. Wan, J. Li, P. Chen, J. Dai, Thirty-eight Conference on Neural Information Y. Yao, R. Xu, Z. Qi, W. Zhao, L. Shen, J. Lu, Processing Systems Datasets a… — Least-to-most prompting enables complex reasoning in large language models
  105. E. Zhang, V. Zhu, N. Saphra, A. Kleiman, B. L. Edelman, M. Tambe, S. M. Kakade, and E. Malach. Transcendence: Generative models can outperform the experts that train them, 2024b. URL https://arxiv.org/ abs/2406.11741 . ↗
  106. H. Zhang, J. Da, D. Lee, V. Robinson, C. Wu, W. Song, T. Zhao, P. Raja, D. Slack, Q. Lyu, et al. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332, 2024c.
  107. Y. Zhang, S. Mao, T. Ge, X. Wang, A. de Wynter, Y. Xia, W. Wu, T. Song, M. Lan, and F. Wei. Llm as a mastermind: A survey of strategic reasoning with large language models. arXiv preprint arXiv:2404.01230, 2024d.
  108. Y. Zhang, Y. Yuan, and A. C.-C. Yao. On the diagram of thought, 2024e. URL https://arxiv.org/abs/2409.10038 . — [arXiv:2409.10038](https://arxiv.org/abs/2409.10038) ↗
  109. Z. Zhang, A. Zhang, M. Li, and A. Smola. Automatic chain of thought prompting in large language models, 2022. URL https://arxiv.org/abs/2210.03493 . ↗
  110. Z. Zhang, A. Zhang, M. Li, H. Zhao, G. Karypis, and A. Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023.
  111. Z. Zhao, W. S. Lee, and D. Hsu. Large language models as commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36, 2024.
  112. A. Zhou, K. Yan, M. Shlapentokh-Rothman, H. Wang, and Y.-X. Wang. Language agent tree search unifies reasoning acting and planning in language models, 2024. URL https://arxiv.org/abs/2310.04406 . ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schultz-2024-mastering-board-games-external-internal/