言語モデルの思考連鎖的推論における探索戦略の動的変化

Summary

The authors study how large language models choose the next step when reasoning in the chain-of-thought style. The material consists of multi-step arithmetic word problems in which some of the conditions are not needed for the answer. Two possible strategies were tested. Under the first, the model implicitly traces the shortest path from the conditions to the answer. Under the second, it relies on surface cues: the similarity of a sentence to the question, the absence of negation, and the sentence's position in the text. First, a single extra sentence carrying such a cue was added to GSM8K problems, and the authors counted how often GPT-3.5-turbo mentioned it in its reasoning. Then, on a synthetic problem set, GPT-3.5-turbo and GPT-4 were given part of the correct chain, and the authors observed what the model chose next: the needed condition or the "decoy". It turned out that the strategy changes over the course of solving. At the start, the models more often follow surface cues, and the closer they get to the answer, the more often they choose steps on the shortest path. GPT-4 chooses shortest-path steps more often than GPT-3.5, even at early stages. The work shows at which stage CoT reasoning is most easily derailed by irrelevant information.

Key points

• Data for experiment 1. GSM8K (about 1K test problems) was taken, and six variants with an additional distractor sentence were created: GSM8K-overlap / not-overlap (the subject matches or does not match the subject of the question), GSM8K-pos / neg (the sentence without or with "not"), GSM8K-head / tail (the sentence at the beginning or at the end of the problem statement).
• GPT-3.5-turbo results. The model mentioned the distractor in 59.5% of cases versus 39.2% for overlap / not-overlap, in 27.0% versus 24.3% for pos / neg, and in 39.2% versus 27.0% for head / tail. Similarity of the sentence to the question has the strongest effect.
• Experiment 2. 100 synthetic problems about numbers of apples were created in a "formal language"; in each, the shortest path has length 3. The models received the first steps of the correct chain, and the distractor had the same subject as the question.
• Dynamics. In GPT-3.5 and GPT-4, the share of steps following a surface cue decreases with each step, while the share of shortest-path steps increases. GPT-4 finds the shortest path more confidently, even when there are still many options. The authors attribute this to the model's larger size.
• Limitations. Only models of the GPT family were tested. The conclusions are drawn solely from the models' behavior, without analysis of their internal representations. The paper was published at NLP2024.