基于解题思维机制的大语言模型多项选择阅读理解增强框架

Summary

Large language models often underperform on multiple-choice reading comprehension (MCRC). Both direct answering and generic chain-of-thought prompting tend to lock onto one option early, without carefully linking passage, question, and candidates. The authors propose a framework built on two human-like Question-Solving Thought (QST) styles: step-by-step, which reasons toward the answer and briefly checks the alternatives, and option-by-option, which judges each candidate's validity and then compares them.

To build QST rationales, they prompt ChatGPT with the gold answer. For the harder option-wise rationales, ChatGPT is guided by GPT-4-written exemplars. The rationales are then used in two ways:

• Self Activate (SA): zero-shot or 1-shot prompting, suited to stronger LLMs.
• Distill from Teacher (DT): fine-tuning weaker models on the teacher's reasoning traces rather than its answers.

Across six MCRC benchmarks, the methods report large average gains in accuracy and strict accuracy. This suggests that task-specific reasoning templates help more than generic CoT for option-based comprehension.

Key points

• Evaluation covers six datasets: RACE-M, RACE-H, CosmosQA, ReClor, LogiQA, and LogiQA2.0. Compared with existing baselines, average accuracy rises by 22.61% and average strict accuracy (accuracy-hard) by 28.95%.
• Two rationale sets of 9,866 items each (D_QST-STEP and D_QST-OPTION) were built from the training splits of RACE, CosmosQA, ReClor, LogiQA, and LogiQA2.0.
• Option-by-option rationales are produced in two stages. GPT-4 first writes zero-shot exemplars. ChatGPT then generates the full set from 5 sampled exemplars, at temperature 1.0, with outputs regenerated until every option is addressed.
• SA uses either a zero-shot instruction or a single exemplar from the QST sets.
• DT trains only on the teacher's intermediate reasoning, not its answer-selection ability, to reduce overfitting to ChatGPT.