📄 Reliable, Adaptable, and Attributable Language Models with Retrieval

Summary

This position paper by Asai et al. contends that retrieval-augmented LMs should succeed purely parametric LMs as the main design for future language models. It contains no new experiments, datasets, or models. Instead, it surveys prior work and lists five weaknesses of parametric models: hallucinated facts, outputs that are hard to verify, trouble removing private or copyrighted training data, costly adaptation to new domains or time periods, and very large model sizes. The authors argue that querying an external datastore at inference time reduces these problems, because knowledge can be cited, updated, or removed without retraining.

They also explain why adoption remains limited. Current systems work mainly on knowledge-heavy tasks such as QA. Simply prepending retrieved passages to an off-the-shelf LM couples the two components only loosely. Open infrastructure for scaling is also scarce. The proposed roadmap calls for rethinking relevance and datastore content, designing more integrated architectures and training methods, and building scalable open-source systems.

Key points

• The paper names five parametric-LM weaknesses (W1–W5): factual errors, weak verifiability, hard data opt-out, expensive adaptation, and huge parameter counts. It cites evidence that scaling alone barely helps with long-tail knowledge.
• It identifies three barriers (C1–C3):
  • Retrieval that relies narrowly on semantic or lexical similarity.
  • Shallow retriever–LM interaction in prompt-concatenation RAG.
  • Insufficient training and inference infrastructure.
• It traces the field's history from DrQA through jointly trained systems (REALM, RAG, RETRO) to today's training-free RAG built on top of models like GPT-3.
• The roadmap includes:
  • Going past input augmentation to designs that blend retrieved information into the output distribution or merge it within the model's internal layers.
  • Training the retriever and LM jointly.
  • Adding retrieval during pre-training.
  • Growing datastores toward trillions of tokens.
• For infrastructure, it highlights FAISS, embedding compression and binary indexes, and possibly specialized CPU-heavy hardware. It notes that LangChain, LlamaIndex, and DSPy exist but argues the field still lacks a standard pipeline and benchmark beyond basic RAG.