♫ MuSiQue: Multihop Questions via Single-hop Question Composition

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.tacl-1.31.pdf

Summary

Existing multihop QA benchmarks such as HotpotQA can often be solved through shortcuts. A model can reach the right answer without linking evidence across the reasoning steps, which the paper calls disconnected reasoning. Trivedi and colleagues build a dataset that rules this out by design. Instead of having crowdworkers write multihop questions directly, they assemble them from the bottom up. Single-hop questions taken from existing Wikipedia-based datasets are chained together, and a pair is kept only if the later question cannot be answered without the earlier question's answer. The pipeline also limits train–test overlap at the level of each single-hop question and adds hard distractor paragraphs. The result is MuSiQue-Ans, with about 25K questions of 2–4 hops. A second version, MuSiQue-Full, pairs each answerable question with an unanswerable counterpart. Compared with HotpotQA and 2WikiMultihopQA, both versions are harder and much less open to shortcut exploitation. The work gives the field a more reliable test of genuine multi-step reasoning, and its gold decompositions are useful for building interpretable models.

Key points

• MuSiQue-Ans has about 25K questions with 2–4 hops, arranged in six reasoning-graph shapes. MuSiQue-Full has about 50K questions forming answerable/unanswerable contrast pairs, and the unanswerable versions are made by removing the paragraphs that contain the bridge entity.
• The gap between humans and the best model on answer F1 is about 27 points, compared with roughly 10 for HotpotQA and 5 for 2WikiMultihopQA, which is about a 3× increase. On supporting-paragraph identification a 14-point gap remains, while models already exceed humans on the other two datasets.
• A single-hop model loses about 30 F1 points compared with its performance on earlier datasets, which shows the dataset is much harder to cheat via disconnected reasoning.
• Ablations show that each pipeline step matters:
  • Disconnection filter: without it, both multihop and shortcut-based models score much higher.
  • Leakage-aware split: with a random split, a single-paragraph model reaches 85.1 answer F1.
  • Distractors: paragraphs taken from the supporting contexts of other questions in the pool ("positive distractors"), with 20 per question, are harder than distractors retrieved from full Wikipedia.
• Training with gold decompositions improves answer F1 by about 9 points and support F1 by about 7 points, but performance still falls short of human level. Code and data are at github.com/stonybrooknlp/musique.