📎 Retrieval-augmented generation

About 16 minutes to read · every paper named here has its own page in this library

What it is

A language model (LM) learns facts by compressing its training text into its weights. That store has three weaknesses:

• It is frozen at training time, so later events are missing.
• It is lossy, so rare facts get blurred or invented, a failure usually called hallucination.
• It is opaque, so a reader cannot tell where a claim came from.

Retrieval-augmented generation addresses all three with an open-book step. When a question arrives, a search component looks up passages in a document collection, and the generator writes its answer with those passages in view.

Surveys describe the same motivation in nearly identical terms. Large language models (LLMs) invent facts, rely on stale training data, and give answers that cannot be traced to a source. Retraining them on new or domain-specific material is costly, while swapping or updating a document store is cheap (Gao et al. 2023; Chua; & Li 2024). The external store works as an updatable, non-parametric memory. It can hold text, code, images or graph data, and it can be consulted at different points in generation (Zhao et al. 2024).

For a newcomer, the simplest picture is a two-part system. A search engine finds a handful of relevant paragraphs. A text generator then reads the question together with those paragraphs and writes the answer, ideally pointing back to the paragraphs it used.

The term was coined for a specific model built for knowledge-intensive tasks. It joined a neural retriever to a sequence-to-sequence (seq2seq) generator and fine-tuned the query encoder and the generator jointly (Lewis et al. 2020). Today "RAG" names the whole family of systems that put retrieved evidence into a generator. RAG is closely tied to question answering (QA), to information retrieval, and to natural language generation more broadly.

How it works

The three components

A RAG system has three parts.

• An index: a collection of documents split into retrievable units. These units are usually called passages or chunks.
• A retriever: it scores units against the query and returns the top k.
• A reader or generator: it produces the output conditioned on the query and the retrieved units.

The basic "retrieve-read" pipeline runs in five steps (Gao et al. 2023):

1. Raw files are converted to plain text and cut into chunks.
2. The chunks are embedded and stored in a vector database.
3. At query time, the question is encoded with the same embedding model.
4. The nearest chunks are fetched.
5. The chunks and the question are pasted together into the generator's prompt.

Sparse and dense retrieval

Sparse retrievers represent text as weighted bags of terms. The standard is BM25, from Robertson and colleagues in the 1990s, which builds on the inverse-document-frequency weighting of Spärck Jones (1972). Until 2020, open-domain QA retrieval was dominated by such keyword methods (Karpukhin et al. 2020).

Dense retrievers instead map queries and passages to dense vectors in a shared space. Dense Passage Retrieval (DPR) uses a dual-encoder design built on BERT, a pretrained Transformer text encoder. One BERT encoder embeds questions and a second embeds passages, and relevance is their inner product (Karpukhin et al. 2020):

s(q, p) = E_Q(q) · E_P(p)

In this formula:

• q is the query and p a passage.
• E_Q and E_P are the question and passage encoders. Each returns a fixed-length vector; in DPR this is the 768-dimensional [CLS] output of BERT-base.
• The symbol · is the dot product.

Because E_P(p) does not depend on the query, every passage can be encoded once, offline. The passages are then searched with a maximum-inner-product index such as FAISS, a library for fast vector similarity search. The DPR authors tested other similarity functions and found no meaningful gain over the dot product (Karpukhin et al. 2020).

A middle ground is late interaction. ColBERT keeps one vector per token. It scores a query against a document by summing, over query tokens, the best match among the document's tokens (Khattab & Zaharia 2020):

s(q, d) = Σ_i max_j  q_i · d_j

Here q_i is the embedding of the i-th query token and d_j that of the j-th document token. The max picks each query token's closest document token. This keeps documents precomputable while recovering much of the token-level matching that full cross-attention provides (Khattab & Zaharia 2020).

The RAG formula

The original RAG model treats the retrieved passage as a latent variable and marginalises over the top k candidates (Lewis et al. 2020). In its "sequence" form:

p(y | x) ≈ Σ_{z ∈ top-k(x)}  p_η(z | x) · Π_{i=1..N} p_θ(y_i | x, z, y_1..y_{i-1})

The symbols mean the following:

• x is the input and y the output sequence of N tokens.
• z ranges over the k passages the retriever returns.
• p_η(z | x) is the retriever's probability for passage z. It is a softmax over the dense scores s(x, z), with retriever parameters η.
• p_θ is the generator, with parameters θ. It predicts each token y_i from the input, the passage and the tokens already produced.

In the "token" variant, the sum moves inside the product, so a different passage can support each output token. The marginal likelihood is differentiable in η, so the query encoder can be trained from the generator's loss without passage labels. The passage index, by contrast, stays fixed (Siriwardhana et al. 2023).

Where retrieval enters

Retrieved content can enter a generator in four ways (Zhao et al. 2024):

• as extra input text;
• as latent representations injected partway through the network;
• as output logits that are mixed with the model's own;
• as a signal that alters or skips generation steps.

Prompt-level injection dominates current practice with large models. In-context retrieval augmentation simply places documents in front of an unmodified LM's input (Ram et al. 2023). REPLUG works with models available only through an API. It runs each passage through the frozen model separately and averages the next-token distributions, weighted by retrieval similarity (Shi et al. 2023).

History

Retrieve-then-read question answering predates neural generators. Chen et al. (2017) paired a Wikipedia search engine with a neural reading-comprehension model. Concurrently with the 2020 work described next, the kNN-LM of Khandelwal et al. (2020) showed that a language model could improve by consulting stored examples at prediction time.

The year 2020 consolidated the modern form. REALM pre-trained a masked language model jointly with a learned retriever over Wikipedia (Guu et al. 2020).

DPR showed that a dense retriever needs no special pre-training. It needs only question–passage pairs and in-batch negatives, which use other questions' correct passages from the same batch as negatives. DPR's results were (Karpukhin et al. 2020):

• On Natural Questions, its top-5 retrieval accuracy was 65.2%, against 42.9% for BM25.
• With an extractive reader, it reached 41.5% exact match (EM), against 33.3% for the earlier dense system ORQA.

Lewis and colleagues then combined a DPR-style retriever with a generator based on BART, a pretrained seq2seq model, and named the result RAG (Lewis et al. 2020). The KILT benchmark, released the same year, found that a single dense retriever plus a seq2seq generator in this style outperformed specialised systems on fact checking, open QA and dialogue (Petroni et al. 2020).

Fusion-in-Decoder (FiD) followed. It encodes each passage separately with T5, a pretrained text-to-text Transformer, and lets the decoder attend over all of them (Izacard & Grave 2021). The 770M-parameter version reached 51.4% EM on NaturalQuestions while reading 100 passages. Even with BM25 retrieval, the same 770M model beat an 11B-parameter closed-book T5, 44.1% to 36.6% EM (Izacard & Grave 2021).

Retrieval also proved useful in dialogue. Human judges found that retrieval-augmented chatbots cut hallucinated replies by more than 60% relative to models without retrieval (Shuster et al. 2021).

RETRO scaled the idea into language-model training itself, retrieving from a database of trillions of tokens (Borgeaud et al. 2021). Atlas showed that a jointly trained retriever and reader could learn knowledge tasks from few examples (Izacard et al. 2022).

After ChatGPT's release in late 2022, the emphasis moved from retrieval-enhanced pre-training to prompting frozen models at inference time. It then moved again, to combining retrieval with fine-tuning (Gao et al. 2023). Representative systems of this period include:

• Demonstrate-Search-Predict (DSP, 2022) (Khattab et al. 2022);
• IRCoT (2022) (Trivedi et al. 2022);
• FLARE (2023), for active retrieval during generation (Jiang et al. 2023);
• Self-RAG (2023) (Asai et al. 2023).

From 2024, graph-structured indexes became a distinct line (Edge et al. 2024; Peng et al. 2024), as did corrective and adaptive pipelines (Yan et al. 2024; Jeong et al. 2024).

Methods

Architectures

Gao et al. sort systems into three paradigms (Gao et al. 2023):

• Naive RAG: a single retrieve-read pass.
• Advanced RAG: adds pre-retrieval steps such as finer segmentation, metadata and query rewriting, and post-retrieval steps.
• Modular RAG: swaps components in and out and allows non-sequential flows.

A complementary grouping has four classes (Chua; & Li 2024):

• RAG frameworks: REALM, RAG, FiD, RETRO, FLARE, Self-RAG.
• Joint RAG learning: Atlas, RA-DIT.
• Retriever learning: DPR, Contriever.
• Pre- or post-retrieval techniques, such as query expansion and compression.

Training the retriever

Much of the progress came from training recipes rather than new architectures.

• RocketQA examined MS MARCO, a large passage-ranking collection built from search-engine queries. It found that many top-ranked but unlabelled passages there are actually relevant. Mined hard negatives, meaning high-scoring passages assumed to be irrelevant, are therefore often false. RocketQA responded with cross-batch negatives, cross-encoder denoising and pseudo-labelling (Qu et al. 2021).
• RocketQAv2 trained the retriever and the re-ranker together through a shared listwise distillation objective. This objective aligns the retriever's score distribution over each candidate list with the re-ranker's (Ren et al. 2021).
• Condenser reshaped pre-training so that the [CLS] vector carries sequence-level meaning (Gao & Callan 2021). Its successor coCondenser added a corpus-level contrastive loss and matched RocketQA with ordinary batch sizes (Gao & Callan 2022).

When labels are absent, generation can substitute for them.

• GPL writes synthetic queries for target-domain passages and learns from cross-encoder score margins (Wang et al. 2022).
• Promptagator prompts a large LM with a few in-task examples to create training queries (Dai et al. 2022).
• HyDE skips training altogether. An LLM writes a hypothetical answer document, and its embedding is used to search real documents (Gao et al. 2022).

The generator itself can also supervise the retriever. REPLUG's LSR variant steers the retriever toward passages that reduce the frozen LM's perplexity, a measure of how poorly the model predicts the text (Shi et al. 2023). RAG-end2end updates the passage encoder and periodically rebuilds the index, which proved central to domain transfer (Siriwardhana et al. 2023).

Chunking and indexing

The retrievable unit matters. Classic open-QA setups cut Wikipedia into non-overlapping 100-word passages (Izacard & Grave 2021). Advanced pipelines add sliding windows, finer segmentation and metadata (Gao et al. 2023). A systematic module-by-module study treated chunking, the embedding model and the vector database as separate choices to tune (Wang et al. 2024).

Graph-based RAG replaces flat chunks with entities and relations. LightRAG uses an LLM to extract a knowledge graph at indexing time. It then retrieves at two levels, specific entities and broad themes (Guo et al. 2024). Graph RAG targets corpus-wide, query-focused summarisation, where no single chunk holds the answer (Edge et al. 2024). A survey organises this line into graph-based indexing, graph-guided retrieval and graph-enhanced generation (Peng et al. 2024).

Query formulation and re-ranking

The user's words are often a poor search query. Rewrite-Retrieve-Read trains a small rewriter with reinforcement learning, using a frozen reader's accuracy as the reward (Ma et al. 2023).

After a cheap first stage, a stronger model often re-orders the candidates. Options include:

• ColBERT, which can re-rank BM25 output far faster than full BERT rankers (Khattab & Zaharia 2020);
• cross-encoder re-rankers, trained jointly with the retriever in RocketQAv2 (Ren et al. 2021);
• the unsupervised UPR, which scores each passage by how likely a pretrained LM is to generate the question from it (Sachan et al. 2022);
• Poly-encoder re-ranking, used for dialogue (Shuster et al. 2021).

In the best-practices study, monoT5 re-ranking appeared in the highest-quality configuration, and fusing sparse and dense retrieval helped cut latency (Wang et al. 2024). Fusing BM25 with a domain dense retriever also gave the most reliable results in medicine (Xiong et al. 2024).

Iterative, adaptive and self-critical retrieval

Multi-hop questions defeat single-shot retrieval because the evidence for the second hop depends on the first. Several methods address this:

• IRCoT alternates one chain-of-thought sentence with one new search. This raises retrieval recall by 11–21 points over question-only retrieval (Trivedi et al. 2022).
• ITER-RETGEN uses a whole draft answer as the next query (Shao et al. 2023).
• DSP composes search and prediction steps as short programs over a frozen LM and ColBERTv2 (Khattab et al. 2022).

Other methods decide whether to retrieve at all.

• Adaptive-RAG uses a small complexity classifier to route each query to no retrieval, single-step retrieval or multi-step retrieval (Jeong et al. 2024).
• Self-RAG adds reflection tokens to the vocabulary. The model learns to emit them to signal four things: whether to retrieve, whether a passage is relevant, whether the output is supported, and how useful the output is. Decoding can then trade citation precision against completeness without retraining (Asai et al. 2023).
• CRAG places a light evaluator between retrieval and generation. It filters weak passages and falls back to web search when retrieval looks wrong (Yan et al. 2024).
• Chain-of-Note has the reader annotate each document before answering. It can therefore answer "unknown" when neither the documents nor its memory suffice (Yu et al. 2024).

Beyond text

The same pattern extends to many modalities and domains:

• code generation and summarisation (Parvez et al. 2021);
• premise selection in formal theorem proving (Yang et al. 2023);
• images in open QA (Chen et al. 2022);
• textual graphs, where retrieval shrank WebQSP inputs from about 100,627 tokens to 610 on average (He et al. 2024).

Domain adaptations include biomedicine, where Self-BioRAG adapted Self-RAG to clinical questions (Jeong et al. 2024), and law (Louis et al. 2023; Wiratunga et al. 2024). Related topics are code and programming, multimodal learning, legal AI and dialogue systems.

Datasets and benchmarks

Open-domain QA collections are the traditional testbed: Natural Questions, TriviaQA, SQuAD-Open and WebQuestions, scored by EM (Karpukhin et al. 2020; Izacard & Grave 2021). Multi-hop sets such as HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC probe iterative retrieval (Trivedi et al. 2022). Retriever quality is usually measured on MS MARCO and on BEIR-style zero-shot suites, which test retrievers on many unseen domains without in-domain training (Khattab & Zaharia 2020; Dai et al. 2022).

KILT aligned eleven datasets across five task types to a single Wikipedia snapshot and attached provenance to every answer. Its "KILT scores" credit a correct answer only when the supporting pages are also retrieved. Baseline values on this stricter measure stayed low (Petroni et al. 2020).

Later benchmarks target RAG-specific failure modes.

Evaluating faithfulness and attribution

Answer accuracy does not show whether an answer follows from its evidence. Ragas scores faithfulness without reference answers. An LLM splits the answer into statements and checks each against the retrieved context, and the score is the share of statements that are supported. Ragas also scores answer relevance and context relevance.

Ragas was checked against human pairwise preferences on WikiEval (Es et al. 2023):

The faithfulness scheme reduces to a simple ratio:

F = |S_supported| / |S|

Here S is the set of statements extracted from the answer, and S_supported is the subset that the context entails.

Other approaches attack attribution from different angles.

• eRAG judges retrieval by feeding each document alone to the system's own LLM and scoring the output against the gold answer. This correlated better with end-to-end quality than provenance labels did (Salemi & Zamani 2024).
• Self-RAG reports citation accuracy as a primary outcome (Asai et al. 2023).
• RAGTruth trains compact detectors that flag unsupported spans (Niu et al. 2024).

Several authors warn that surface metrics mislead. Exact match can seriously underrate correct LLM answers (Shao et al. 2023). In legal QA, reasonable overlap scores hid answers that often contained invented facts (Louis et al. 2023). More on measurement practice is collected under evaluation and benchmarks.

People and institutions

The DPR paper did much to make dense retrieval the default first stage (Karpukhin et al. 2020). It was led by Vladimir Karpukhin, with Barlas Oguz, Sewon Min, Patrick Lewis, Danqi Chen and Wen-tau Yih among its authors.

Patrick Lewis also led the paper that named RAG (Lewis et al. 2020) and co-authored KILT and Atlas (Petroni et al. 2020; Izacard et al. 2022). Gautier Izacard and Edouard Grave introduced Fusion-in-Decoder (Izacard & Grave 2021). Kelvin Guu and colleagues introduced REALM (Guu et al. 2020), and Sebastian Borgeaud and colleagues introduced RETRO (Borgeaud et al. 2021).

Akari Asai and Hannaneh Hajishirzi developed Self-RAG (Asai et al. 2023). Wen-tau Yih, Mike Lewis, Luke Zettlemoyer, Weijia Shi and Sewon Min contributed REPLUG (Shi et al. 2023). Yih and Mike Lewis also worked on unsupervised re-ranking with UPR (Sachan et al. 2022).

Luyu Gao and Jamie Callan produced Condenser, coCondenser and HyDE (Gao & Callan 2021; Gao & Callan 2022; Gao et al. 2022). Omar Khattab, with Matei Zaharia, created ColBERT (Khattab & Zaharia 2020). Khattab later built DSP with Christopher Potts and others (Khattab et al. 2022).

The RocketQA line came from Yingqi Qu, Ruiyang Ren, Jing Liu, Wayne Xin Zhao, Hua Wu and Haifeng Wang (Qu et al. 2021; Ren et al. 2021). The work was a collaboration between Baidu and Renmin University of China. Haifeng Wang and Hua Wu are at Baidu, and Ren and Zhao are at Renmin University (Ren et al. 2021).

Institutions linked to these researchers include:

• the University of Washington: Asai, Hajishirzi, Zettlemoyer, Min;
• Carnegie Mellon University: Callan, Gao;
• Princeton University: Chen;
• the Allen Institute for Artificial Intelligence: Min, Asai, Hajishirzi;
• Meta: Yih, Mike Lewis, Zettlemoyer.

Industrial laboratories behind other systems on this page include:

• Google: REALM and Promptagator (Guu et al. 2020; Dai et al. 2022);
• Google DeepMind: RETRO (Borgeaud et al. 2021);
• Microsoft: Graph RAG and Rewrite-Retrieve-Read (Edge et al. 2024; Ma et al. 2023).

Chinese universities also contributed:

• Peking University produced two of the surveys discussed above (Zhao et al. 2024; Peng et al. 2024).
• Tsinghua University took part in ITER-RETGEN (Shao et al. 2023).
• Fudan University carried out the best-practices study (Wang et al. 2024).

Open questions

Robustness to bad evidence

Retrieval can hurt. RGB found that models tolerate some noise but rarely refuse when evidence is missing, and handle multi-document questions poorly. They also tend to believe retrieved misinformation. Only models scoring above 70% closed-book accuracy were tested on counterfactual documents. Their accuracy dropped from 77–91% without documents to 9–17% once they were fed counterfactual ones (Chen et al. 2023).

Distracting but on-topic passages lower Llama-2's accuracy. Even top-1 web results hurt it on some multi-hop benchmarks. Fine-tuning on mixed relevant and irrelevant contexts restores robustness (Yoran et al. 2023).

Oddly, one study found that adding random unrelated documents raised accuracy by up to 35%. The experiments covered open-domain QA with 7B readers such as Llama2-7b and the Contriever retriever. This suggests that retrievers tuned for human search may not suit LLM readers (Cuconasu et al. 2024).

Position, length and quantity

More context is not monotonically better.

• In medical QA, accuracy rose log-linearly with the number of snippets. It also depended on where in the prompt the relevant snippet appeared (Xiong et al. 2024).
• Gold passages work best near the query (Cuconasu et al. 2024).
• In dialogue, retrieving more documents improved perplexity but increased hallucination as rated by humans (Shuster et al. 2021).
• FiD, by contrast, kept improving up to 100 passages (Izacard & Grave 2021).

When to stop retrieving remains unsettled.

Parametric versus retrieved knowledge

It is still unclear when a model draws on its own memory and when it uses the passage (Shi et al. 2023). Deciding whether to retrieve at all is a learned guess. Adaptive-RAG's complexity classifier was only about 53–55% accurate, which leaves a clear gap to an oracle router (Jeong et al. 2024).

Faithful attribution

Systems that retrieve correctly can still cite wrongly. KILT's provenance-gated scores stayed low (Petroni et al. 2020). Span-level hallucination detection remains far from solved even with dedicated training data (Niu et al. 2024). LLM-as-judge metrics inherit the biases of the judging model (Es et al. 2023).

Cost and scale

Iterative methods add model calls.

• Chain-of-Note raised average decoding time from about 0.61 s to about 12.02 s per example, roughly twentyfold (Yu et al. 2024).
• IRCoT needs one LM call per reasoning step (Trivedi et al. 2022).
• End-to-end retriever training must re-encode the whole corpus (Siriwardhana et al. 2023).
• Graph indexes must scale from thousands to billions of entities (Peng et al. 2024).

Domain transfer

Retrievers trained on general web data lose quality in specialised fields (Wang et al. 2022). Legal retrieval still placed fewer than half of the relevant articles in the top five (Louis et al. 2023). The Self-BioRAG authors found that open-domain recipes such as Self-RAG misjudge relevance on clinical questions (Jeong et al. 2024).

The central practical question for the field is how to build retrievers and generators that adapt cheaply to new collections and can say reliably when they do not know.

The library in figures