Elamur Research⌕
☰
home / papers / 2025 / Mitigating Lost-in-Retrieval Problems in Retrieval Augmented Multi-Hop Question Answering

Mitigating Lost-in-Retrieval Problems in Retrieval Augmented Multi-Hop Question Answering

Paper · 2025 · Rongzhi Zhu, Xiangyu Liu, Zequn Sun et al.

Summary

In retrieval-augmented multi-hop QA, LLMs often split a complex question into sub-questions. Later sub-questions then refer to an earlier result through a pronoun such as "this author" instead of naming the entity. The authors call this failure "lost-in-retrieval." Without the key entity, retrieval for these sub-questions degrades and the reasoning chain collapses. In an empirical check on MuSiQue, 2Wiki and HotpotQA, the second sub-question was retrieved markedly worse than the first.

To address this, they propose ChainRAG. It first builds a sentence graph whose edges come from three sources: shared named entities extracted with spaCy, embedding similarity and textual proximity. It then works through the sub-questions in order: retrieve sentences, answer the current sub-question, and use that answer to fill in the missing entity in the next one. Finally, all retrieved sentences and intermediate answers are merged into one final response.

Tested with GPT4o-mini, Qwen2.5-72B and GLM-4-Plus on LongBench versions of the three datasets, ChainRAG beat the baselines in both accuracy and efficiency. It needs no fine-tuning and no costly LLM-based knowledge-graph construction.

Key points

  • Empirical study: 300 random examples per dataset. Recall@2 for the second sub-question was on average 18.29% lower than for the first, across several chunk sizes.
  • The sentence graph has three edge types:
    • entity co-occurrence (EC), limited to the top-α% key entities ranked by BM25;
    • semantic similarity (SS), linking top-m neighbours using OpenAI text-embedding-3-small;
    • structural adjacency (SA), linking sentences within 3 positions of each other.
  • Two answer-generation variants exist: answer integration (AnsInt) and sub-context integration (CxtInt).
  • A manual review of 600 Qwen2.5-72B test questions found 89.3% correct question decomposition and 79.3% correct entity recovery. Entity recovery errors hurt F1 more than decomposition errors.
  • Using Qwen2.5-72B instead of spaCy for entity recognition gave only small gains on MuSiQue, so spaCy remains the more economical choice. The main stated limitation is higher cost than NaiveRAG, though ChainRAG is still cheaper than LongRAG and HippoRAG.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanjing UniversityNanjing · China74 works · 7,814 citations🎓University of California, MercedMerced · United States11 works · 425 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Question answering · 441 worksInformation retrieval & search · 473 works

Cite

Cite as Rongzhi Zhu, Xiangyu Liu, Zequn Sun, Yiwei Wang and Wei Hu. 2025. Mitigating Lost-in-Retrieval Problems in Retrieval Augmented Multi-Hop Question Answering. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1089

@inproceedings{zhu2025mitigating,
  title = {{Mitigating Lost-in-Retrieval Problems in Retrieval Augmented Multi-Hop Question Answering}},
  author = {Rongzhi Zhu, Xiangyu Liu, Zequn Sun, Yiwei Wang and Wei Hu},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1089},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1089}
}

References

34 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. Longbench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Associat… find
  2. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph RAG approach to query-focused summarization. CoRR, abs/2404.16130. find
  3. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. CoRR, abs/2312.10997.
  4. Michael R. Glass, Gaetano Rossiello, Md. Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. 2022. Re2g: Retrieve, rerank, generate. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, N…
  5. Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. HippoRAG: Neurobiologically inspired long-term memory for large language models. CoRR, abs/2405.14831.
  6. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021.
  7. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing A multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, pages 6609–6625.
  8. Ziyan Jiang, Xueguang Ma, and Wenhu Chen. 2024. LongRAG: Enhancing retrieval-augmented generation with long-context llms. CoRR, abs/2406.15319. — [arXiv:2406.15319](https://arxiv.org/abs/2406.15319) find
  9. Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Informatio…
  10. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. BGE m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. CoRR, abs/2402.03216.
  11. Jiawei Li, Yizhe Yang, Yu Bai, Xiaofeng Zhou, Yinghao Li, Huashan Sun, Yuhang Liu, Xingpeng Si, Yuhao Ye, Yixiao Wu, Yiguan Lin, Bin Xu, Ren Bowen, Chong Feng, Yang Gao, and Heyan Huang. 2024a. Fundamental capabilities of large language models and their applications in domain scenarios: A survey. In… find
  12. Xavier Daull, Patrice Bellot, Emmanuel Bruno, Vincent Martin, and Elisabeth Murisasco. 2023. Complex QA and language models hybrid architectures, survey. CoRR, abs/2302.09051. — [arXiv:2302.09051](https://arxiv.org/abs/2302.09051) find
  13. Shilong Li, Yancheng He, Hangyu Guo, Xingyuan Bu, Ge Bai, Jie Liu, Jiaheng Liu, Xingwei Qu, Yangguang Li, Wanli Ouyang, Wenbo Su, and Bo Zheng. 2024b. Graphreader: Building graph-based agent to Huajun Chen. 2024. Large knowledge model: Perspectives and challenges. Data Intelligence, 6. enhance long-… find
  14. Zheng Liu, Chenyuan Wu, Ninglu Shao, Shitao Xiao, Chaozhuo Li, and Defu Lian. 2024. Lighter and better: Towards flexible context adaptation for retrieval augmented generation. CoRR, abs/2409.15699. — [arXiv:2409.15699](https://arxiv.org/abs/2409.15699) find
  15. Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2024a. Reasoning on graphs: Faithful and interpretable large language model reasoning. In The Twelfth International Conference on Learning Representations, ICLR 2024. find
  16. Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Seyed Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y. Zhao. 2023. Dr.icl: Demonstration-retrieved in-context learning. CoRR, abs/2305.14128. — [arXiv:2305.14128](https://arxiv.org/abs/2305.14128) find
  17. Xindi Luo, Zequn Sun, Jing Zhao, Zhe Zhao, and Wei Hu. 2024b. KnowLA: Enhancing parameter-efficient finetuning with knowledgeable adaptation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 7153–… find
  18. Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. 2024. Is your LLM outdated? benchmarking llms & alignment algorithms for time-sensitive knowledge. CoRR, abs/2404.08700. — [arXiv:2404.08700](https://arxiv.org/abs/2404.08700) find
  19. Jesse Mu, Xiang Li, and Noah D. Goodman. 2023. Learning to compress prompts with gist tokens. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023.
  20. OpenAI. 2023. GPT-4 technical report. abs/2303.08774.
  21. CoRR, Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. 2024. RAPTOR: recursive abstractive processing for tree-organized retrieval. In The Twelfth International Conference on Learning Representations, ICLR 2024. OpenReview.net.
  22. Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 9248–…
  23. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi-hop questions via single-hop question composition. Trans. Assoc. Comput. Linguistics, 10:539–554. find
  24. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A…
  25. Ziting Wang, Haitao Yuan, Wei Dong, Gao Cong, and Feifei Li. 2024. CORAG: A cost-constrained retrieval optimization system for retrieval-augmented generation. CoRR, abs/2411.00744. — [arXiv:2411.00744](https://arxiv.org/abs/2411.00744) find
  26. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul F. Christiano. 2021. Recursively summarizing books with human feedback. CoRR, abs/2109.10862.
  27. Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2024a. RE-COMP: improving retrieval-augmented lms with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations, ICLR 2024. find
  28. Ziwei Xu, Sanjay Jain, and Mohan S. Kankanhalli. 2024b. Hallucination is inevitable: An innate limitation of large language models. CoRR, abs/2401.11817.
  29. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei… find
  30. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2…
  31. Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, R… — ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
  32. Jintian Zhang, Cheng Peng, Mengshu Sun, Xiang Chen, Lei Liang, Zhiqiang Zhang, Jun Zhou, Huajun Chen, and Ningyu Zhang. 2024. Onegen: Efficient one-pass unified generation and retrieval for llms. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4088–4119. find
  33. Methods Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang. 2023. Multilingual machine translation with large language models: Empirical results and analysis. CoRR, abs/2304.04675.
  34. Ziyuan Zhuang, Zhiyang Zhang, Sitao Cheng, Fangkai Yang, Jia Liu, Shujian Huang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2024. Efficientrag: Efficient retriever for multi-hop question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Proces… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2025-mitigating-lost-retrieval-problems-retrieval/