Elamur Research⌕
☰
home / papers / 2025 / RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructions

RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructions

Paper · 2025 · Wanlong Liu, Junying Chen, Ke Ji et al.

Summary

Retrieval-augmented LLMs are usually tuned on narrow QA sets such as NQ or TriviaQA. As a result, they cope poorly when retrieved passages only partly help, do not help at all, or must be combined across several documents. The authors present RAG-Instruct, a recipe for generating varied RAG training data from any corpus. GPT-4o writes an instruction and an answer grounded in source documents, and unrelated passages are mixed in as noise. Diversity comes from two sources. First, five query–document paradigms vary how useful the documents are and how many of them matter. Second, "Instruction Simulation" uses knowledge-intensive prompts from existing instruction collections as templates, so generated questions vary in task type, wording and difficulty. Applied to Wikipedia, the method produced a 40K-example dataset. Llama models fine-tuned on it beat several RAG baselines across 11 benchmarks, with the biggest gains in partially helpful and unhelpful retrieval settings. This matters because a general-purpose, corpus-agnostic RAG instruction set improves robustness to imperfect retrievers without task-specific tuning.

Key points

  • Five paradigms (r0–r4): Useless Doc, Single-Doc Support, Multi-Doc Support, Single-Doc Answer and Multi-Doc Answer. Each is defined by document usefulness and the count of useful documents (|D*|).
  • Instruction Simulation: exemplar questions come from ShareGPT, Alpaca, WizardLM-70K, Lmsys-chat-1M and SlimOrca. GPT-4o first filters them for knowledge-intensive prompts.
  • Preliminary study (accuracy %): RAG-Instruct (Llama3-8b) was compared with Self-RAG, RQ-RAG, ChatQA-1.5 and ChatQA-2.0.
    • On the TriviaQA "midhelp" subset it scored 72.6, against at most 54.9 for the baselines.
    • On the "helpless" subset it scored 40.5, against at most 21.4.
    • On HotpotQA midhelp it scored 42.2, against at most 24.0.
  • Comparison with other datasets: relative to Llama3.1-8B-Instruct, training Llama3.1-8B on RAG-Instruct improved TQA by +6.6, HotpotQA by +12.8, ARC by +9.6 and CFQA by +4.1. Every other dataset compared (LongAlpaca, SQuAD2.0, NarrativeQA, RAG-12000, Self-RAG Data, RQ-RAG Data) lost accuracy on CFQA.
  • Availability and limitations: the code is released at github.com/FreedomIntelligence/RAG-Instruct. The authors acknowledge two limitations: the paradigms are coarse-grained, and the GPT-generated data may contain errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chinese University of Hong Kong, ShenzhenShenzhen · China47 works · 2,596 citations🎓University of Electronic Science and Technology of ChinaChengdu · China31 works · 2,012 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksInformation retrieval & search · 473 works

Cite

Cite as Wanlong Liu, Junying Chen, Ke Ji, Li Zhou, Wenyu Chen and Benyou Wang. 2025. RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructions. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.192

@inproceedings{liu2025rag,
  title = {{RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructions}},
  author = {Wanlong Liu, Junying Chen, Ke Ji, Li Zhou, Wenyu Chen and Benyou Wang},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.192},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.192}
}

References

52 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Parishad BehnamGhader, Santiago Miret, and Siva Reddy. 2022. Can retriever-augmented language models reason? the blame game between the retriever and the language model. arXiv preprint arXiv:2212.09146. find
  2. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544.
  3. Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024. Rq-rag: Learning to refine queries for retrieval augmented generation. arXiv preprint arXiv:2404.00610. find
  4. Yukang Chen, Shaozuo Yu, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023. Long alpaca: Long-context instruction-following models. https://github.com/dvlab-research/ LongLoRA. ↗
  5. Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6279– 6…
  6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  7. Yixiong Fang, Tianran Sun, Yuling Shi, and Xiaodong Gu. 2025. Attentionrag: Attention-guided context pruning in retrieval-augmented generation. arXiv preprint arXiv:2503.10720. — [arXiv:2503.10720](https://arxiv.org/abs/2503.10720) find
  8. Pengbin Feng and Xuhui Peng. 2014. A note on monge– kantorovich problem. Statistics & Probability Letters, 84:204–211. find
  9. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.
  10. Jiani Guo, Zuchao Li, Jie Wu, Qianren Wang, Yun Li, Lefei Zhang, hai zhao, and Yujiu Yang. 2025. Tom: Leveraging tree-oriented mapreduce for long-context reasoning in large language models. In The 2025 Conference on Empirical Methods in Natural Language Processing. find
  11. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR.
  12. Tsun hin Cheung and Kin Man Lam. 2023. Factllama: Optimizing instruction-following language models with external knowledge for automated fact-checking. 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 846–853. find
  13. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781.
  14. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609– 6625.
  15. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328.
  16. Or Honovich et al. 2022. Unnatural instructions: Tuning language models with synthetic data. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5021–5035. Association for Computational Linguistics. find
  17. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research.
  18. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1–43.
  19. Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2024. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics:…
  20. Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983.
  21. Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. 2024. Flashrag: A modular toolkit for efficient retrieval-augmented generation research. arXiv preprint arXiv:2405.13576. find
  22. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
  23. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611.
  24. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistic…
  25. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
  26. H.Y. Leong, Y. Gao, S. Ji, Y. Zhang, and U. Pamuksuz. 2024. Efficient fine-tuning of large language models for automated medical documentation. In 2024 4th IEEE International Conference on Digital Society and Intelligent Systems (DSInS), Sydney, Australia. find
  27. H.Y. Leong and Y. Wu. 2024. Why should next-gen llm multi-agent systems move beyond fixed architectures to dynamic, input-driven graphs? SSRN Electronic Journal. find
  28. Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. 2023. Ra-dit: Retrieval-augmented dual instruction tuning. arXiv preprint arXiv:2310.01352.
  29. Aofan Liu, SONG Shiyuan, Cehao Yang, and Yiyan Qi. 2025a. Beyond function-level search: Repository-aware dual-encoder code retrieval with adversarial verification. In Knowledgeable Foundation Models at ACL 2025. find
  30. Aofan Liu, Yuguo Yin, Hongjian Xing, Zhen Li, and Yiyan Qi. 2025b. Md3r: Minimizing data distribution discrepancies to tackle inconsistencies in multilingual query-code retrieval. In Knowledgeable Foundation Models at ACL 2025. find
  31. Wanlong Liu, Enqi Zhang, Li Zhou, Dingyi Zeng, Shaohuan Cheng, Chen Zhang, Malu Zhang, and Wenyu Chen. 2024a. A compressive memory-based retrieval approach for event argument extraction. arXiv preprint arXiv:2409.09322. find
  32. Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Mohammad Shoeybi, and Bryan Catanzaro. 2024b. Chatqa: Building gpt-4 level conversational qa models. arXiv preprint arXiv:2401.10225.
  33. Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svyatkovskiy. 2022. Reacc: A retrieval-augmented code completion framework. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6227–6240.
  34. Xuan Lu, Sifan Liu, Bochao Yin, Yongqi Li, Xinghao Chen, Hui Su, Yaohui Jin, Wenjun Zeng, and Xiaoyu Shen. 2025. Multiconir: Towards multi-condition information retrieval. Preprint, arXiv:2503.08046. find
  35. Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5303–5315.
  36. Seiji Maekawa, Hayate Iso, Sairam Gurajada, and Nikita Bhutani. 2024. Retrieval helps or hurts? a deeper dive into the efficacy of retrieval augmentation to language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… find
  37. Sebastian Riedel. 2020. How context affects language models’ factual predictions. In Automated Knowledge Base Construction.
  38. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1– 16. IEEE. find
  39. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Ling…
  40. Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. find
  41. Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9248–9274.
  42. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… find
  43. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR.
  44. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391. find
  45. Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2024. Replug: Retrieval-augmented black-box language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
  46. Swaroop Mishra et al. 2022. Natural instructions: Benchmarking generalization in instruction following. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5021–5035. Association for Computational Linguistics. find
  47. Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, et al. 2023. Orca 2: Teaching small language models how to reason. arXiv preprint arXiv:2311.11045.
  48. Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. Asqa: Factoid questions meet long-form answers. arXiv preprint arXiv:2204.06092.
  49. Wenting Tan, Dongxiao Chen, Jieting Xue, Zihao Wang, and Taijie Chen. 2025. Teaching-inspired integrated prompting framework: A novel approach for enhancing reasoning in large language models. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, pages 827… find
  50. Rishi Taori et al. 2023. Alpaca: A 175b-parameter model for instruction following. ArXiv preprint arXiv:2303.11366.
  51. Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H Miller, and Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multi-hop questions via single-hop question composition. find
  52. OpenAI. 2023. Sharegpt: scale instruction-following Https://openai.com/research/sharegpt. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2025-rag-instruct-boosting-llms-diverse/