📄 Self-Guide:一种基于自我规划的大语言模型推理增强方法

Summary

Chain-of-thought prompting asks a large language model to plan a solution and execute it in a single pass. Drawing on Sweller's Cognitive Load Theory, the authors argue that this double burden can overload the model and produce hallucinated or unproductive reasoning. Their method, Self-Guide, splits planning from answering and needs no fine-tuning or external tools. In a zero-shot setting, the model first writes a "Guideline" for the question, covering relevant background knowledge, an interpretation of the question, and a step-by-step plan. In parallel, it produces an ordinary Zero-Shot-CoT answer. A template then asks the model to revisit that answer in light of the Guideline, fixing errors and refining the final response. The approach was tested with GPT-3.5-turbo-1106 on four kinds of reasoning, where it beat Zero-Shot and CoT baselines. Guidelines produced by GPT-3.5 also gave large gains to the weaker LLaMA2 chat models. This suggests a stronger model's plans can be handed to a smaller one, a cheap route to better reasoning without any training.

Key points

• Benchmarks:
  • MMLU (knowledge understanding, 500 random samples)
  • StrategyQA (multi-hop QA, full set)
  • Date Understanding from BIG-bench (temporal reasoning, full set)
  • CLUTRR (kinship reasoning, 500 random samples)
• Gains: on GPT-3.5-turbo-1106, the average improvement over standard chain-of-thought was about 2%.
• Transfer to smaller models: Guidelines from GPT-3.5 lifted LLaMA2-7B-Chat and LLaMA2-13B-Chat by over 10%.
• Evaluation and decoding:
  • Metric: accuracy via lowercase exact string matching, with task-specific answer extraction (e.g., MM/DD/YYYY date patterns for Date).
  • Decoding temperature: 0.2.
• Availability: code and data are released at github.com/NEUIR/Self-Guide (Northeastern University, with Tsinghua University and Beijing Language and Culture University co-authors).