Elamur Research⌕
☰
home / papers / 2024 / Puzzle Solving using Reasoning of Large Language Models: A Survey

Puzzle Solving using Reasoning of Large Language Models: A Survey

Paper · 2024 · Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos et al.

Summary

This EMNLP 2024 survey examines how well large language models handle puzzles and which techniques help them. The authors propose a taxonomy based on the skills a puzzle requires rather than on question format or reasoning type. Rule-based puzzles have explicit rules and are either deterministic (Sudoku, Chess, crosswords, Game of 24) or stochastic (Minesweeper, card games, social deduction games). Rule-less puzzles include riddles, programming puzzles and commonsense reasoning puzzles. Text-free, multimodal and mathematical puzzles are excluded.

The survey groups solving methods into three families: prompting (CoT, Self-Consistency, ToT, XoT and related thought structures), neuro-symbolic translation of puzzles into formal code, and task-specific fine-tuning. It also catalogues the available datasets and benchmarks.

The main finding is that LLMs still fall well short of human performance, especially on multi-step logical inference. Richer prompting structures help, but they do not close the gap. The authors identify missing benchmarks and methods, which makes the survey a useful map for future work on LLM reasoning.

Key points

  • Benchmarks catalogued include:
    • riddles: RiddleSense, BrainTeaser, BiRdQA, CC-Riddle, PUZZLEQA, MARB
    • programming: P3
    • commonsense: LatEval, True Detective, DetectBench
    • rule-based: BoardgameQA
  • Reported results:
    • On Game of 24, Self-Refine achieved a success rate 13% higher than CoT.
    • On DetectBench, GPT-4 was the strongest model at 61.6% accuracy, and a new Detective Thinking Prompt gave the best prompting results.
  • Fine-tuning transfers poorly. Models tuned on CSQA scored lower on RiddleSense than models tuned on RiddleSense itself.
  • Gaps and format effects:
    • Few datasets exist for stochastic rule-based puzzles and programming puzzles.
    • Neuro-symbolic approaches are rarely applied.
    • Multiple-choice formats are noticeably easier for models than free-text answers.
  • Scope and tracking: the review covers about four years of work, mainly from ACL, EMNLP, NAACL, NeurIPS, ICLR and arXiv. Ongoing progress is tracked at puzzlellms.github.io.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Technical University of AthensAthens · Greece8 works · 461 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos and Giorgos Stamou. 2024. Puzzle Solving using Reasoning of Large Language Models: A Survey. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.646

@inproceedings{giadikiaroglou2024puzzle,
  title = {{Puzzle Solving using Reasoning of Large Language Models: A Survey}},
  author = {Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos and Giorgos Stamou},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.646},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.646}
}

References

106 references, 36 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Forest Agostinelli, Stephen Marcus McAleer, Alexander Shmakov, and Pierre Baldi. 2019. Solving the rubik’s cube with deep reinforcement learning and search. Nature Machine Intelligence, 1:356 – 363. find
  2. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Tachard Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Z. Chen, Eric Chu, J. Clark, Laurent El Shafey, Yanping Huang, Kathleen S. Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… — PaLM 2 Technical Report
  3. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. ArXiv, abs/2302.04…
  4. Qiming Bao, Gaël Gendron, Alex Yuxuan Peng, Wanjun Zhong, Neşet Özkan Tan, Yang Chen, Michael Witbrock, and Jiamou Liu. 2023. A systematic evaluation of large language models on out-of-distribution logical reasoning tasks. ArXiv, abs/2310.09430. — [arXiv:2310.09430](https://arxiv.org/abs/2310.09430) find
  5. Houda Nait El Barj and Theophile Sautory. 2024. Reinforcement learning from llm feedback to counteract goal misgeneralization. find
  6. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. 2023. Graph of thoughts: Solving elaborate problems with large language models. find
  7. Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Guangyuan Piao, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwaśniewski, Jürgen Müller, Lukas Gianinazzi, Ales Kubicek, Hubert Niewiadomski, Aidan O’Mahony, Onur Mutlu, and Torsten Hoefler. 2024. Demystifying chains, trees, and g… find
  8. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. Piqa: Reasoning about physical commonsense in natural language. ArXiv, abs/1911.11641.
  9. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, C… — Language Models are Few-Shot Learners
  10. Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Ya… find
  11. Murray Campbell, A.Joseph Hoane, and Feng hsiung Hsu. 2002. Deep blue. Artificial Intelligence, 134(1):57–83. find
  12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. In Annual Meeting of the Association for Computational Linguist… find
  13. Banghao Chen, Zhaofeng Zhang, Nicolas Langren’e, and Shengxin Zhu. 2023. Unleashing the potential of prompt engineering in large language models: a comprehensive review. ArXiv, abs/2310.14735.
  14. Juntao Chen. 2022. Different algorithms to solve a rubik’s cube. Journal of Physics: Conference Series, 2386(1):012018. find
  15. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Aleth… — Evaluating Large Language Models Trained on Code
  16. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. ArXiv, abs/2211.12588. find
  17. Eric C. Chi and Kenneth Lange. 2013. Techniques for solving sudoku puzzles. find
  18. Yew Ken Chia, Vernon Toh Yan Han, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. 2024. Puzzlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns. find
  19. Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2023. A survey of chain of thought reasoning: Advances, frontiers and future. ArXiv, abs/2309.15402. find
  20. Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. ArXiv, abs/2205.09712. find
  21. Maksym Del and Mark Fishel. 2022. True detective: A deep abductive reasoning benchmark undoable for gpt-3 and challenging for gpt-4. In STARSEM. find
  22. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding.
  23. Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Ming-Jie Ma, Wei Zhang, Si Qin, S. Rajmohan, Qingwei Lin, and Dongmei Zhang. 2023. Everything of thoughts: Defying the law of penrose triangle for thought generation. ArXiv, abs/2311.04254. find
  24. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey on in-context learning.
  25. Avia Efrat, Uri Shaham, Dan Kilman, and Omer Levy. 2021. Cryptonite: A cryptic crossword benchmark for extreme ambiguity in language. ArXiv, abs/2103.01242. find
  26. Jiazhan Feng, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, and Weizhu Chen. 2023a. Language models can be logical solvers. ArXiv, abs/2311.06158. — [arXiv:2311.06158](https://arxiv.org/abs/2311.06158) find
  27. Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Henry Mguni, Yali Du, and Jun Wang. 2023b. Chessgpt: Bridging policy learning and language modeling. ArXiv, abs/2306.09200. find
  28. Peter A. Flach and Antonis C. Kakas. 2000. Abductive and inductive reasoning: background and issues. find
  29. Yao Fu, Hao-Chun Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. Complexity-based prompting for multi-step reasoning. ArXiv, abs/2210.00720.
  30. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. Pal: Program-aided language models. ArXiv, abs/2211.10435. find
  31. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. find
  32. Deepanway Ghosal, Vernon Toh Yan Han, Chia Yew Ken, and Soujanya Poria. 2024. Are language models puzzle prodigies? algorithmic puzzles unveil serious challenges in multimodal reasoning. find
  33. Zhouhong Gu, Zihan Li, Lin Zhang, Zhuozhi Xiong, Sihang Jiang, Xiaoxuan Zhu, Shusen Wang, Zili Wang, Jianchen Wang, Haoning Ye, Wenhao Huang, Yikai Zhang, Hongwei Feng, and Yanghua Xiao. 2023. Go beyond the obvious: Probing the gap of informal reasoning ability between humanity and llms by detective… find
  34. Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. 2023. Suspicion-agent: Playing imperfect information games with theory of mind aware gpt-4. ArXiv, abs/2309.17277. — [arXiv:2309.17277](https://arxiv.org/abs/2309.17277) find
  35. Akshat Gupta. 2023. Are chatgpt and gpt-4 good poker players? - a pre-flop analysis. ArXiv, abs/2308.12466. — [arXiv:2308.12466](https://arxiv.org/abs/2308.12466) find
  36. Chenghao Huang, Yanbo Cao, Yinlong Wen, Tao Zhou, and Yanru Zhang. 2024. Pokergpt: An end-to-end lightweight solver for multi-player texas hold’em via large language model. ArXiv, abs/2401.06781. — [arXiv:2401.06781](https://arxiv.org/abs/2401.06781) find
  37. Jie Huang and Kevin Chen-Chuan Chang. 2022. Towards reasoning in large language models: A survey. ArXiv, abs/2212.10403.
  38. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023a. Large language models cannot self-correct reasoning yet. ArXiv, abs/2310.01798.
  39. Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ramachandran. 2023. Boardgameqa: A dataset for natural language reasoning with contradictory information. ArXiv, abs/2306.07934. — [arXiv:2306.07934](https://arxiv.org/abs/2306.07934) find
  40. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. Unifiedqa: Crossing format boundaries with a single qa system. In Findings.
  41. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916.
  42. Richard E. Korf. 1997. Finding optimal solutions to rubik’s cube using pattern databases. In AAAI/IAAI. find
  43. Saurabh Kulshreshtha, Olga Kovaleva, Namrata Shivagunde, and Anna Rumshisky. 2022. Down and across: Introducing crossword-solving as a new nlp benchmark. ArXiv, abs/2205.10442. find
  44. Yihuai Lan, Zhiqiang Hu, Lei Wang, Yang Wang, De-Yong Ye, Peilin Zhao, Ee-Peng Lim, Hui Xiong, and Hao Wang. 2023. Llm-based agent society investigation: Collaboration and confrontation in avalon gameplay. ArXiv, abs/2310.14985. find
  45. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. ArXiv, abs/1909.11942.
  46. Bin Lei, Pei-Hung Lin, Chunhua Liao, and Caiwen Ding. 2023. Boosting logical reasoning in large language models through a new framework: The graph of thought. ArXiv, abs/2308.08614. — [arXiv:2308.08614](https://arxiv.org/abs/2308.08614) find
  47. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Annual Meeting of the Association for Comput…
  48. Yinghao Li, Haorui Wang, and Chao Zhang. 2023. Assessing logical puzzle solving in large language models: Insights from a minesweeper case study. ArXiv, abs/2311.07387. find
  49. Shulin Huang, Shirong Ma, Yinghui Li, Mengzuo Huang, Wuhe Zou, Weidong Zhang, and Haitao Zheng. 2023b. Lateval: An interactive llms evaluation benchmark with incomplete information from lateral thinking puzzles. ArXiv, abs/2308.10855. find
  50. Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee, and Xiang Ren. 2021. Riddlesense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge. In Findings. find
  51. Adam Ishay, Zhun Yang, and Joohyung Lee. 2023. Leveraging large language models to generate answer set programs. ArXiv, abs/2307.07699. — [arXiv:2307.07699](https://arxiv.org/abs/2307.07699) find
  52. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yuexin Zhang. 2023a. Evaluating the logical reasoning ability of chatgpt and gpt-4. ArXiv, abs/2304.03439.
  53. Yifan Jiang, Filip Ilievski, and Kaixin Ma. 2023. Brainteaser: Lateral thinking puzzles for large language models. In Conference on Empirical Methods in Natural Language Processing. find
  54. Hanmeng Liu, Zhiyang Teng, Ruoxi Ning, Jian Liu, Qiji Zhou, and Yuexin Zhang. 2023b. Glore: Evaluating logical reasoning of large language models. ArXiv, abs/2310.09107. — [arXiv:2310.09107](https://arxiv.org/abs/2310.09107) find
  55. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55:1 – 35.
  56. Wentao Liu, Hanglei Hu, Jie Zhou, Yuyang Ding, Junsong Li, Jiayi Zeng, Mengliang He, Qin Chen, Bo Jiang, Aimin Zhou, and Liang He. 2023c. Mathematical language models: A survey. find
  57. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
  58. Jieyi Long. 2023. Large language model guided tree-of-thought. ArXiv, abs/2305.08291. find
  59. Man Luo, Shrinidhi Kumbhar, Ming shen, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya, and Chitta Baral. 2023. Towards logiglue: A brief survey and a benchmark for analyzing logical reasoning capabilities of language models. ArXiv, abs/2310.00836.
  60. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. ArX… find
  61. Smaragda Markaki and Costas Panagiotakis. 2022. Jigsaw puzzle solving techniques and applications: a survey. The Visual Computer, 39:4405 – 4421. find
  62. Stephen McAleer, Forest Agostinelli, Alexander Shmakov, and Pierre Baldi. 2018. Solving the rubik’s cube without human knowledge. find
  63. Arindam Mitra and Chitta Baral. 2015. Learning to automatically solve logic grid puzzles. In Conference on Empirical Methods in Natural Language Processing. find
  64. Shentong Mo and Miao Xin. 2023. Tree of uncertain thoughts reasoning for large language models. ArXiv, abs/2309.07694. — [arXiv:2309.07694](https://arxiv.org/abs/2309.07694) find
  65. David A. Noever and Ryerson Burdick. 2021. Puzzle solving without search or human knowledge: An unnatural language approach. ArXiv, abs/2109.02797. — [arXiv:2109.02797](https://arxiv.org/abs/2109.02797) find
  66. Theo X. Olausson, Alex Gu, Benjamin Lipkin, Cedegao Zhang, Armando Solar-Lezama, Josh Tenenbaum, and Roger Levy. 2023. Linc: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers. In Conference on Empirical Methods in Natural Language Processing.
  67. OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake… — GPT-4 Technical Report
  68. Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023a. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. ArXiv, abs/2305.12295.
  69. Jaromir Savelka, Arav Agarwal, Christopher Bogart, and Majd Sakr. 2023. Large language models (gpt) struggle to answer multiple-choice questions about code. find
  70. Tal Schuster, A. Kalyan, Oleksandr Polozov, and Adam Tauman Kalai. 2021. Programming puzzles. ArXiv, abs/2106.05784. — [arXiv:2106.05784](https://arxiv.org/abs/2106.05784) find
  71. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. 2017. Mastering chess and shogi by self-play with a general reinforcement learning alg…
  72. Helmut Simonis. 2005. Sudoku as a constraint problem. find
  73. Chris Studholme. 2001. Minesweeper as a constraint satisfaction problem. find
  74. Roxana Szomiu and Adrian Groza. 2021. A puzzle-based dataset for natural language inference. ArXiv, abs/2112.05742. — [arXiv:2112.05742](https://arxiv.org/abs/2112.05742) find
  75. Kyo Takano. 2023. Self-supervision is all you need for solving rubik’s cube. find
  76. Liangming Pan, Michael Stephen Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2023b. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. ArXiv, abs/2308.03188. find
  77. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. ArXiv, abs/1811.00937.
  78. Julien Pourcel, Cédric Colas, Pierre-Yves Oudeyer, and Laetitia Teodorescu. 2023. Aces: Generating diverse programming puzzles with autotelic language models and semantic descriptors. ArXiv, abs/2310.10692. — [arXiv:2310.10692](https://arxiv.org/abs/2310.10692) find
  79. Yongqi Tong, Yifan Wang, Dawei Li, Sizhe Wang, Zi Lin, Simeng Han, and Jingbo Shang. 2023. Eliminating reasoning via inferring with planning: A new framework to guide llms’ non-linear thinking. ArXiv, abs/2310.12342. — [arXiv:2310.12342](https://arxiv.org/abs/2310.12342) find
  80. Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022. Reasoning with language model prompting: A survey. ArXiv, abs/2212.09597.
  81. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  82. Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
  83. Josh Rozner, Christopher Potts, and Kyle Mahowald. 2021. Decrypting cryptic crosswords: Semantically complex wordplay puzzles as a target for nlp. ArXiv, abs/2104.08620. find
  84. Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Seyed Mehran Kazemi, Najoung Kim, and He He. 2023. Testing the general deductive reasoning capacity of large language models using ood examples. ArXiv, abs/2305.15269.
  85. Gladys Tyen, Hassan Mansoor, Peter Chen, Tony Mak, and Victor Carbune. 2023. Llms cannot find reasoning errors, but can correct them! ArXiv, abs/2311.08516. — [arXiv:2311.08516](https://arxiv.org/abs/2311.08516) find
  86. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023a. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Annual Meeting of the Association for Computational Linguistics. find
  87. Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. 2023b. Avalon’s game of thoughts: Battle against deception through recursive contemplation. ArXiv, abs/2310.01320. — [arXiv:2310.01320](https://arxiv.org/abs/2310.01320) find
  88. Weiqi Wang, Tianqing Fang, Wenxuan Ding, Baixuan Xu, Xin Liu, Yangqiu Song, and Antoine Bosselut. 2023c. Car: Conceptualization-augmented reasoner for zero-shot commonsense question answering. In Conference on Empirical Methods in Natural Language Processing. find
  89. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. ArXiv, abs/2203.11171.
  90. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903.
  91. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. 2022. Large language models are better reasoners with self-verification. In Conference on Empirical Methods in Natural Language Processing.
  92. Fan Xu, Yunxiang Zhang, and Xiao-Yi Wan. 2022. Cc-riddle: A question answering dataset of chinese character riddles. ArXiv, abs/2206.13778. — [arXiv:2206.13778](https://arxiv.org/abs/2206.13778) find
  93. Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. 2023a. Are large language models really good logical reasoners? a comprehensive evaluation and beyond.
  94. Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. 2023b. Exploring large language models for communication games: An empirical study on werewolf. ArXiv, abs/2309.04658.
  95. Sen Yang, Xin Li, Leyang Cui, Li Bing, and Wai Lam. 2023a. Neuro-symbolic integration brings causal and reliable reasoning proofs. ArXiv, abs/2311.09802. find
  96. Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and E. Cambria. 2023b. Logical reasoning over natural language as knowledge representation: A survey. ArXiv, abs/2303.12023.
  97. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. ArXiv, abs/2305.10601. find
  98. Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. 2023a. Natural language reasoning, a survey. find
  99. Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jiajun Chen. 2023b. Towards better chain-of-thought prompting strategies: A survey. ArXiv, abs/2310.04959. — [arXiv:2310.04959](https://arxiv.org/abs/2310.04959) find
  100. Kamyar Zeinalipour, Tommaso laquinta, Asya Zanollo, Giovanni Angelini, Leonardo Rigutini, Marco Maggini, and Marco Gori. 2023a. Italian crossword generator: Enhancing education through interactive word puzzles. find
  101. Kamyar Zeinalipour, Mohamed Saad, Marco Maggini, and Marco Gori. 2023b. Arabicros: Ai-powered arabic crossword puzzle generation for educational applications. In Proceedings of ArabicNLP 2023. Association for Computational Linguistics.
  102. Yunxiang Zhang and Xiaojun Wan. 2021. Birdqa: A bilingual dataset for question answering on tricky riddles. ArXiv, abs/2109.11087. — [arXiv:2109.11087](https://arxiv.org/abs/2109.11087) find
  103. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alexander J. Smola. 2022. Automatic chain of thought prompting in large language models. ArXiv, abs/2210.03493.
  104. Jingmiao Zhao and Carolyn Jane Anderson. 2023. Solving and generating npr sunday puzzles with large language models. find
  105. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. ArXiv, abs/2211.01910.
  106. Andrea Zugarini, Kamyar Zeinalipour, Surya Sai Kadali, Marco Maggini, Marco Gori, and Leonardo Rigutini. 2024. Clue-instruct: Text-based clue generation for educational crossword puzzles. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/giadikiaroglou-2024-puzzle-solving-reasoning-large-language/