Elamur Research⌕
☰
home / papers / 2026 / HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions

HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions

Paper · 2026 · Zhiyu Shen, Jiyuan Liu, Yunhe Pang et al.

Summary

Building multi-hop QA (MHQA) datasets by hand is expensive. Automatic generators usually need curated evidence or knowledge graphs, or they produce questions that one document alone can answer. HopWeaver is a pipeline that works from a raw text corpus with no human in the loop. It produces two question types: bridge questions, which chain facts through an intermediate entity, and comparison questions, which contrast a shared attribute of two entities.

For bridge questions, an LLM picks a bridge entity in a randomly sampled document and writes a retrieval query. Candidate documents are gathered with a modified MMR score that favours query relevance and penalises overlap with the source and among the candidates. A fine-tuned reranker then reorders them. Sub-questions grounded in each document are merged, and a Polisher module repairs or discards flawed outputs.

The authors evaluated on English Wikipedia using LLM-as-judge, answerability/difficulty, and evidence-accessibility checks. The generated questions match or exceed human-annotated benchmarks at lower cost. This makes the approach useful for building benchmarks in domains without annotated data.

Key points

  • Reranker ablation (bridge questions): synthesis success rose from 70.1% with standard dense retrieval to 75.3% with MMR plus the fine-tuned reranker. Average attempts fell from 1.59 to 1.17.
  • Polisher effect: gains are largest for weaker generators. For GLM-4-9B, bridge multi-hop validity went from 84.5% to 89.8% and the average score from 3.71 to 3.87. Improvements for Gemini-2.5-flash were marginal.
  • Efficiency audit (100 successful samples per type, Gemini-2.5-flash): most bridge rejections happen at sub-question generation (29 cases, about 78% of bridge failures). A question costs about 7.6 API calls on average, retries included.
  • Comparison benchmarks: HotpotQA, 2WikiMultiHopQA and MuSiQue. Experiments are English-only, though the authors argue the pipeline is language-agnostic given multilingual embeddings. This is relevant for Ukrainian corpora.
  • Code: publicly available at github.com/Zh1yuShen/HopWeaver. An extension to n-hop questions via recursive generation is described in the appendix.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sun Yat-sen UniversityGuangzhou · China53 works · 9,185 citations

Also on the byline: Hong Kong Metropolitan University.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Zhiyu Shen, Jiyuan Liu, Yunhe Pang, Yanghui Rao, Fu Lee Wang and Jianxing Yu. 2026. HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1295

@inproceedings{shen2026hopweaver,
  title = {{HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions}},
  author = {Zhiyu Shen, Jiyuan Liu, Yunhe Pang, Yanghui Rao, Fu Lee Wang and Jianxing Yu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1295},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1295}
}

References

46 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya… — LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
  2. Jaime G. Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 335–336. find
  3. Ruirui Chen, Weifeng Jiang, Chengwei Qin, Ishaan Singh Rawal, Cheston Tan, Dongkyu Choi, Bo Xiong, and Bo Ai. 2024. LLM-based multi-hop question answering with knowledge graph integration in evolving environments. In Findings of the Association for Computational Linguistics: EMNLP, pages 14438–14451… find
  4. Zhi Chen, Qiguang Chen, Libo Qin, Qipeng Guo, Haijun Lv, Yicheng Zou, Hang Yan, Kai Chen, and Dahua Lin. 2025. What are the essential factors in crafting effective long context multi-hop instruction datasets? insights and best practices. In Proceedings of the 63rd Annual Meeting of the Association f…
  5. Yi Cheng, Siyao Li, Bang Liu, Ruihui Zhao, Sujian Li, Chenghua Lin, and Yefeng Zheng. 2021. Guiding the growth: Difficulty-controllable question generation through step-by-step rewriting. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internat… find
  6. David Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 15607–15631. find
  7. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1342–1352.
  8. Zichu Fei, Qi Zhang, Tao Gui, Di Liang, Sirui Wang, Wei Wu, and Xuanjing Huang. 2022. CQG: A simple and effective controlled generation framework for multi-hop question generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 6896–6906. find
  9. Emilio Ferrara. 2023. Should chatgpt be biased? challenges and risks of bias in large language models. First Monday, 28(11). find
  10. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378. find
  11. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024a. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 6556–6576. find
  12. Weiping Fu, Bifan Wei, Jianxiang Hu, Zhongmin Cai, and Jun Liu. 2024b. QGEval: Benchmarking multidimensional evaluation for question generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11783–11803. find
  13. Shasha Guo, Lizi Liao, Cuiping Li, and Tat-Seng Chua. 2024. A survey on neural question generation: Methods, applications, and prospects. In Proceedings of the 33 International Joint Conference on Artificial Intelligence, pages 8038–8047. find
  14. Xu Guo and Yiqiang Chen. 2024. Generative AI for synthetic data generation: Methods, challenges and the future. CoRR, abs/2403.04190. — [arXiv:2403.04190](https://arxiv.org/abs/2403.04190) find
  15. Jie He, Nan Hu, Wanqiu Long, Jiaoyan Chen, and Jeff Z. Pan. 2024. MINTQA: A multi-hop question answering benchmark for evaluating llms on new and tail knowledge. CoRR, abs/2412.17032. — [arXiv:2412.17032](https://arxiv.org/abs/2412.17032) find
  16. Michael Heilman and Noah A. Smith. 2010. Good question! statistical ranking for question generation. In Proceedings of Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, pages 609– 617. find
  17. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625.
  18. Linmei Hu, Zeyi Liu, Ziwang Zhao, Lei Hou, Liqiang Nie, and Juanzi Li. 2024. A survey of knowledge enhanced pre-trained language models. IEEE Transactions on Knowledge and Data Engineering, 36(4):1413–1430. find
  19. Yizheng Huang and Jimmy Huang. 2024. A survey on retrieval-augmented text generation for large language models. CoRR, abs/2404.10981. — [arXiv:2404.10981](https://arxiv.org/abs/2404.10981) find
  20. Seonjeong Hwang, Yunsu Kim, and Gary Geunbae Lee. 2024. Explainable multi-hop question generation: An end-to-end approach without intermediate question labeling. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, pages 6855– 686… find
  21. Jiajie Jin, Yutao Zhu, Zhicheng Dou, Guanting Dong, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, and Ji-Rong Wen. 2025a. Flashrag: A modular toolkit for efficient retrieval-augmented generation research. In Companion Proceedings of the ACM on Web Conference, pages 737–740. find
  22. Jiajie Jin, Yutao Zhu, Zhicheng Dou, Guanting Dong, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, and Ji-Rong Wen. 2025b. Flashrag: A modular toolkit for efficient retrieval-augmented generation research. In Companion Proceedings of the ACM on Web Conference, pages 737–740. find
  23. Jan-Christoph Klie, Richard Eckart de Castilho, and Iryna Gurevych. 2024. Analyzing dataset annotation quality management in the wild. Computational Linguistics, 50(3):817–866.
  24. Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage Publications. find
  25. Noah Lee, Jiwoo Hong, and James Thorne. 2025. Evaluating the consistency of LLM evaluators. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10650–10659.
  26. Hao Liu, Zhengren Wang, Xi Chen, Zhiyu Li, Feiyu Xiong, Qinhan Yu, and Wentao Zhang. 2025. Ho-prag: Multi-hop reasoning for logic-aware retrieval-augmented generation. In Findings of the Association for Computational Linguistics: ACL, pages 1897–1913. find
  27. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522.
  28. Adian Liusie, Potsawee Manakul, and Mark J. F. Gales. 2024. LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pages 139–151.
  29. Yingzhou Lu, Huazheng Wang, and Wenqi Wei. 2023. Machine learning for synthetic data generation: a review. CoRR, abs/2302.04062. — [arXiv:2302.04062](https://arxiv.org/abs/2302.04062) find
  30. Haohao Luo, Yang Deng, Ying Shen, See-Kiong Ng, and Tat-Seng Chua. 2024. Chain-of-exemplar: Enhancing distractor generation for multimodal educational question generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 7978–7993. find
  31. Alisia Lupidi, Carlos Gemmell, Nicola Cancedda, Jane Dwivedi-Yu, Jason Weston, Jakob Foerster, Roberta Raileanu, and Maria Lomeli. 2025. Source2synth: Synthetic data generation and curation grounded in real data sources. In Proceedings of the Conference on Language Modeling. find
  32. Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt. 2022. A survey on multi-hop question answering and generation. CoRR, abs/2204.09140. — [arXiv:2204.09140](https://arxiv.org/abs/2204.09140) find
  33. Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt. 2024. Multi-hop question answering. Foundations and Trends in Information Retrieval, 17(5):457–586. find
  34. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. Compositional questions do not necessitate multi-hop reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4249–4257.
  35. Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. 2018. Answer-focused and position-aware neural question generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3930–3939.
  36. Yixuan Tang and Yi Yang. 2024. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries. CoRR, abs/2401.15391. llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems.
  37. Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: A preliminary study. In Proceedings of the 6th CCF International Conference on Natural Language Processing and Chinese Computing, pages 662–671. find
  38. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554. find
  39. Nakanyseth Vuth, Gilles Sérasset, and Didier Schwab. 2024. KGAST: From knowledge graphs to annotated synthetic texts. In Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models, pages 43–55. find
  40. Maximilian Wich, Christian Widmer, Gerhard Hagerer, and Georg Groh. 2021. Investigating annotator bias in abusive language datasets. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, pages 1515–1525. find
  41. Ian Wu, Sravan Jayanthi, Vijay Viswanathan, Simon Rosenberg, Sina Pakazad, Tongshuang Wu, and Graham Neubig. 2024. Synthetic multimodal question generation. In Findings of the Association for Computational Linguistics: EMNLP, pages 12960– 12993. find
  42. Brendan Wyse and Paul Piwek. 2009. Generating questions from openlearn study units. In Proceedings of the AIED 2009 Workshop on Question Generation, pages 66–73. find
  43. Zehua Xia, Qi Gou, Bowen Yu, Haiyang Yu, Fei Huang, Yongbin Li, and Cam-Tu Nguyen. 2023. Improving question generation with multi-level content planning. In Findings of the Association for Computational Linguistics: EMNLP, pages 800–814. find
  44. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2…
  45. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V. Chawla, and Xiangliang Zhang. 2025. Justice or prejudice? quantifying biases in llm-as-a-judge. In Proceedings of the 13 International Conference on Learning Representa…
  46. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2026-hopweaver-cross-document-synthesis-high/