Elamur Research⌕
☰
home / papers / 2026 / Fine-Tuning vs. RAG for Multi-Hop Question Answering with Novel Knowledge

Fine-Tuning vs. RAG for Multi-Hop Question Answering with Novel Knowledge

Paper · 2026 · Zhuoyi Yang, Yurun Song, Kyler G. Harris et al.

Summary

The paper asks whether the way new knowledge is supplied to an LLM matters for open-domain multi-hop question answering, especially when the facts postdate the model's pretraining. Earlier comparisons of fine-tuning and retrieval mostly covered single-hop recall and did not fix model size, evaluation format or knowledge recency. The authors compare three injection strategies on three open-source 7B models: unsupervised fine-tuning (continual pretraining on raw text), supervised fine-tuning on question–answer pairs, and RAG over a chunked, vector-indexed corpus. They test on QASC and on a new benchmark of more than 10,000 multi-hop questions built from 2024 Wikipedia event pages. All settings share a multiple-choice accuracy protocol. Continual pretraining barely improves on the base models. RAG helps substantially, and on the 2024 data it roughly doubles accuracy. Supervised fine-tuning gives the best results overall. For practitioners, the takeaway is that retrieval is the practical choice for fresh or compositional knowledge when labelled data is unavailable, while raw-text pretraining alone is a weak option.

Key points

  • Setup: Each of three 7B open-source LLMs is tested in four configurations: base, RAG (with the base model as generator), unsupervised fine-tuning, and supervised fine-tuning.
  • Benchmarks:
    • QASC is a science multi-hop dataset with 8 answer options, so random chance is 1/8.
    • The new 2024 Events dataset has more than 10,000 questions with 4 options, so random chance is 1/4. It is sourced from Wikipedia's "2024 in the United States by month" category.
  • Scoring:
    • The supervised models are scored through a sequence classification head.
    • The base, unsupervised and RAG settings use MMLU-style next-token log-probabilities over the option labels.
    • Accuracy is the metric throughout.
  • Results:
    • On the 2024 Events data, RAG adds about 30 percentage points for every model, more than doubling base accuracy.
    • Unsupervised fine-tuning gives only marginal gains on both datasets.
    • Supervised fine-tuning scores highest overall.
  • Limitations:
    • The evaluation looks only at final-answer accuracy, not the reasoning steps.
    • Results may depend on retriever quality.
    • The dataset was built with LLM help and may contain errors.
    • Code has been promised but not yet released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, IrvineIrvine · United States43 works · 36,565 citations🎓University of California, Santa CruzSanta Cruz · United States38 works · 3,604 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksQuestion answering · 487 works

Cite

Cite as Zhuoyi Yang, Yurun Song, Kyler G. Harris, Iftekhar Ahmed and Ian Harris. 2026. Fine-Tuning vs. RAG for Multi-Hop Question Answering with Novel Knowledge. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.37

@inproceedings{yang2026fine,
  title = {{Fine-Tuning vs. RAG for Multi-Hop Question Answering with Novel Knowledge}},
  author = {Zhuoyi Yang, Yurun Song, Kyler G. Harris, Iftekhar Ahmed and Ian Harris},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.37},
  url = {https://doi.org/10.18653/v1/2026.gem-main.37}
}

References

28 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John Achiam and 1 others. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.
  2. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. Preprint, arXiv:2310.11511.
  3. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency. find
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  5. DeepSeek-AI and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in large language models. arXiv preprint, arXiv:2501.12948.
  6. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. 2020. Shortcut learning in deep neural networks. In Nature Machine Intelligence. find
  7. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  8. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.
  9. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online).
  10. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, and 1 others. 2023a. Mistral 7b. arXiv preprint, arXiv:2310.06825. find
  11. Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023b. Active retrieval augmented generation. Preprint, arXiv:2305.06983.
  12. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547.
  13. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Question answering as global reasoning over semantic abstractions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  14. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. Unifiedqa: Crossing format boundaries with a single qa system. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
  15. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. Preprint, arXiv:1910.11473. find
  16. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-Tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the Advances in Neur…
  17. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advance…
  18. Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2024. Fine-tuning or retrieval? comparing knowledge injection in llms. Preprint, arXiv:2312.05934.
  19. Manu Pande, Shahil Kumar, and Anay Yatin Damle. 2025. When fine-tuning fails: Lessons from ms marco passage ranking. Preprint, arXiv:2506.18535. — [arXiv:2506.18535](https://arxiv.org/abs/2506.18535) find
  20. Keivalya Pandya and Mehfuza Holia. 2023. Automating customer service using langchain: Building custom open-source gpt chatbot for organizations. Preprint, arXiv:2310.05421. — [arXiv:2310.05421](https://arxiv.org/abs/2310.05421) find
  21. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
  22. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. Preprin…
  23. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, and 1 others. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint, arXiv:2307.09288.
  24. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554. find
  25. Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, and Furu Wei. 2025. Chain-of-retrieval augmented generation. Preprint, arXiv:2501.14342. — [arXiv:2501.14342](https://arxiv.org/abs/2501.14342) find
  26. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
  27. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. Preprint, arXiv:2309.07597.
  28. Zhiyuan Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Brussels, B…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-fine-tuning-vs-rag-multi/