Elamur Research⌕
☰
home / papers / 2026 / AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAG

AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAG

Paper · 2026 · Qijie You, Wenkai Yu, Hao Liang et al.

Summary

Popular multi-hop QA benchmarks such as HotpotQA, 2WikiMultihopQA and MuSiQue give only a question and its final answer. This makes it impossible to see where an agentic RAG system goes wrong partway through. The authors also find that some "multi-hop" items are mislabeled, and that the corpora and indices behind these datasets are rarely released. To fix this, they present AgenticRAGTracer, a benchmark built mostly by LLMs and checked by humans. Each question comes with its intermediate hop-level sub-questions.

The pipeline works in stages:

  • samples Wikipedia documents that do not overlap with existing datasets;
  • writes single-fact (atomic) questions;
  • keeps only those that need retrieval, are not too easy and are not unanswerable;
  • chains or combines them into sequential (Inference) or parallel (Comparison) questions.

Further filtering, SPARQL-based disambiguation and manual review follow. Tests on 13 models show weak results, with GPT-5 scoring 22.6% EM on the hardest subset. Hop-level analysis shows that most failures come from badly shaped reasoning chains: agents either stop too early or keep adding unnecessary steps. This kind of step-level insight is missing from evaluations that only score the final answer.

Key points

  • The benchmark contains 1,305 human-verified examples from multiple domains, with no overlap with HotpotQA, 2Wiki or MuSiQue.
  • Questions follow two topologies, Inference (Sequential) and Comparison (Parallel), and each includes golden hop-level traces.
  • It uses the FlashRAG Wikipedia corpus and index for reproducibility; GPT-4o-mini is the default generator in the construction pipeline.
  • Thirteen mainstream LLMs were evaluated, and GPT-5 reaches only 22.6% EM on the hardest portion.
  • Failed runs take noticeably more reasoning steps than successful ones, which suggests that rigid plan execution and the first hop largely determine success.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓University of Science and Technology BeijingBeijing · China8 works · 315 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 worksInformation retrieval & search · 473 works

Cite

Cite as Qijie You, Wenkai Yu, Hao Liang, Zhen Hao Wong and Wentao Zhang. 2026. AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAG. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.66

@inproceedings{you2026agenticragtracer,
  title = {{AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAG}},
  author = {Qijie You, Wenkai Yu, Hao Liang, Zhen Hao Wong and Wentao Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.66},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.66}
}

References

36 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ruichuan An, Sihan Yang, Ming Lu, Renrui Zhang, Kai Zeng, Yulin Luo, Jiajun Cao, Hao Liang, Ying Chen, Qi She, et al. 2024. Mc-llava: Multi-concept personalized vision-language model. arXiv preprint arXiv:2411.11706. — [arXiv:2411.11706](https://arxiv.org/abs/2411.11706) find
  2. Ruichuan An, Sihan Yang, Renrui Zhang, Zijun Shen, Ming Lu, Gaole Dai, Hao Liang, Ziyu Guo, Shilin Yan, Yulin Luo, et al. 2025. Unictokens: Boosting personalized understanding and generation via unified concept tokens. arXiv preprint arXiv:2505.14671. — [arXiv:2505.14671](https://arxiv.org/abs/2505.14671) find
  3. Tianyi Bai, Hao Liang, Binwang Wan, Yanran Xu, Xi Li, Shiyu Li, Ling Yang, Bozhou Li, Yifan Wang, Bin Cui, et al. 2024. A survey of multimodal large language model from a data-centric perspective. arXiv preprint arXiv:2405.16640. — [arXiv:2405.16640](https://arxiv.org/abs/2405.16640) find
  4. Qifeng Cai, Hao Liang, Chang Xu, Tao Xie, Wentao Zhang, and Bin Cui. 2025. Text2sql-flow: A robust sql-aware data augmentation framework for text-to-sql. arXiv preprint arXiv:2511.10192. — [arXiv:2511.10192](https://arxiv.org/abs/2511.10192) find
  5. Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. 2025. Research: Learning to reason with search for llms via reinforcement learning. Preprint, arXiv:2503.19470. find
  6. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  7. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, H… find
  8. Jingwen Deng, Jihao Huang, Zhen Hao Wong, Hao Liang, Quanqing Xu, Bin Cui, and Wentao Zhang. 2025. Data-centric perspectives on agentic retrieval-augmented generation: A survey. Authorea Preprints. find
  9. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024a. Retrieval-augmented generation for large language models: A survey. Preprint, arXiv:2312.10997.
  10. Yunfan Gao, Yun Xiong, Meng Wang, and Haofen Wang. 2024b. Modular rag: Transforming rag systems into lego-like reconfigurable frameworks. Preprint, arXiv:2407.21059. — [arXiv:2407.21059](https://arxiv.org/abs/2407.21059) find
  11. Ziyu Guo, Xinyan Chen, Renrui Zhang, Ruichuan An, Yu Qi, Dongzhi Jiang, Xiangtai Li, Manyuan Zhang, Hongsheng Li, and Pheng-Ann Heng. 2025. Are video models ready as zero-shot reasoners? an empirical study with the mme-cof benchmark. arXiv preprint arXiv:2510.26802. — [arXiv:2510.26802](https://arxiv.org/abs/2510.26802) find
  12. Jie He, Nan Hu, Wanqiu Long, Jiaoyan Chen, and Jeff Z. Pan. 2025. Mintqa: A multi-hop question answering benchmark for evaluating llms on new and tail knowledge. Preprint, arXiv:2412.17032. — [arXiv:2412.17032](https://arxiv.org/abs/2412.17032) find
  13. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Comm…
  14. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan O Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training LLMs to reason and leverage search engines with reinforcement learning. In Second Conference on Language Modeling.
  15. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver…
  16. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural…
  17. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Proc…
  18. Hao Liang, Xiaochen Ma, Zhou Liu, Zhen Hao Wong, Zhengyang Zhao, Zimo Meng, Runming He, Chengyu Shen, Qifeng Cai, Zhaoyang Han, et al. 2025a. Dataflow: An llm-driven framework for unified data preparation and workflow automation in the era of data-centric ai. arXiv preprint arXiv:2512.16676. — [arXiv:2512.16676](https://arxiv.org/abs/2512.16676) find
  19. Hao Liang, Linzhuang Sun, Jingxuan Wei, Xijie Huang, Linkun Sun, Bihui Yu, Conghui He, and Wentao Zhang. 2024. Synth-empathy: Towards high-quality synthetic empathy data. arXiv preprint arXiv:2407.21669. — [arXiv:2407.21669](https://arxiv.org/abs/2407.21669) find
  20. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… find
  21. Julian Schnitzler, Xanh Ho, Jiahao Huang, Florian Boudin, Saku Sugawara, and Akiko Aizawa. 2024. Morehopqa: More than multi-hop reasoning. Preprint, arXiv:2406.13397. — [arXiv:2406.13397](https://arxiv.org/abs/2406.13397) find
  22. Chengyu Shen, Zhen Hao Wong, Runming He, Hao Liang, Meiyi Qiang, Zimo Meng, Zhengyang Zhao, Bohan Zeng, Zhengzhou Zhu, Bin Cui, et al. 2025. Let’s verify math questions step by step. arXiv preprint arXiv:2505.13903. find
  23. Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. 2025. Agentic retrieval-augmented generation: A survey on agentic rag. Preprint, arXiv:2501.09136. — [arXiv:2501.09136](https://arxiv.org/abs/2501.09136) find
  24. Hao Liang, Zhen Hao Wong, Ruitong Liu, Yuhan Wang, Meiyi Qiang, Zhengyang Zhao, Chengyu Shen, Conghui He, Wentao Zhang, and Bin Cui. 2026. Data preparation for large language models. Journal of Computer Science and Technology. find
  25. Linzhuang Sun, Hao Liang, Jingxuan Wei, Bihui Yu, Tianpeng Li, Fan Yang, Zenan Zhou, and Wentao Zhang. 2025. Mm-verify: Enhancing multimodal reasoning with chain-of-thought verification. arXiv preprint arXiv:2502.13383.
  26. Jintao Liang, Gang Su, Huifeng Lin, You Wu, Rui Zhao, and Ziyue Li. 2025b. Reasoning rag via system 1 or system 2: A survey on reasoning agentic retrieval-augmented generation for industry challenges. Preprint, arXiv:2506.10408. find
  27. Yixuan Tang and Yi Yang. 2024. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries. Preprint, arXiv:2401.15391.
  28. Weifeng Lin, Xinyu Wei, Ruichuan An, Tianhe Ren, Tingwei Chen, Renrui Zhang, Ziyu Guo, Wentao Zhang, Lei Zhang, and Hongsheng Li. 2025. Perceive anything: Recognize, explain, caption, and segment anything in images and videos. Preprint, arXiv:2506.05302. find
  29. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554. find
  30. Zheng Liu, Hao Liang, Xijie Huang, Wentao Xiong, Qinhan Yu, Linzhuang Sun, Chong Chen, Conghui He, Bin Cui, and Wentao Zhang. 2024. Synthvlm: High-efficiency and high-quality synthetic data for vision language models. arXiv preprint arXiv:2407.20756. — [arXiv:2407.20756](https://arxiv.org/abs/2407.20756) find
  31. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
  32. Yulin Luo, Ruichuan An, Bocheng Zou, Yiming Tang, Jiaming Liu, and Shanghang Zhang. 2024. Llm as dataset analyst: Subpopulation structure discovery with large language model. In European Conference on Computer Vision, pages 235–252. Springer. find
  33. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
  34. Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. 2024. Graph retrieval-augmented generation: A survey. Preprint, arXiv:2408.08921.
  35. Qingfei Zhao, Ruobing Wang, Dingling Xu, Daren Zha, and Limin Liu. 2025. R-search: Empowering llm reasoning with search via multi-reward reinforcement learning. Preprint, arXiv:2506.04185.
  36. Miao Zheng, Hao Liang, Fan Yang, Haoze Sun, Tianpeng Li, Lingchu Xiong, Yan Zhang, Youzhen Wu, Kun Li, Yanjun Shen, et al. 2024. Pas: Data-efficient plug-and-play prompt augmentation system. arXiv preprint arXiv:2407.06027. — [arXiv:2407.06027](https://arxiv.org/abs/2407.06027) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/you-2026-agenticragtracer-hop-aware-benchmark-diagnosing/