What Is Retrieval-Augmented Generation (RAG)
Key facts
- Retrieval-augmented generation models combine pre-trained parametric and non-parametric memory to generate language.
- Standard pre-trained language models face limitations in accessing and manipulating factual knowledge for knowledge-intensive tasks.
- In the tested RAG architecture, parametric memory is supplied by a pre-trained sequence-to-sequence model while non-parametric memory comes from a dense vector index of Wikipedia.
- Researchers evaluated two RAG setups, including one conditioning on the same passages for a sequence and another using different passages per token.
- RAG models produced more specific, diverse, and factual text compared to a parametric-only sequence-to-sequence baseline.
Limitations of Parametric Language Models
Pre-trained language models retain factual information directly within their internal parameters. However, these models still show limited capacity to access and manipulate knowledge when handling knowledge-intensive tasks. In these situations, their results fall behind specialized task-specific architectures.
The Architecture of RAG Models
Retrieval-augmented generation introduces a framework combining pre-trained parametric memory and non-parametric memory to produce text. The parametric component uses a pre-trained sequence-to-sequence model. Meanwhile, the non-parametric component consists of a dense vector index of Wikipedia that is accessed using a neural retriever.
Researchers developed two distinct formulations for implementing RAG models. One version conditions output on the same retrieved passages throughout the complete generated sequence. The alternative version allows the model to utilize different retrieved passages for each individual token.
Performance on Knowledge-Intensive Tasks
The evaluated RAG models were tested across multiple knowledge-intensive natural language processing benchmarks. The models established state-of-the-art performance on three open domain question answering tasks. They outperformed standard parametric sequence-to-sequence models as well as task-specific architectures built for retrieval and extraction.
Frequently asked questions
What is retrieval-augmented generation?
Retrieval-augmented generation, or RAG, refers to models that combine pre-trained parametric memory with explicit non-parametric memory for language generation. This approach pairs a sequence-to-sequence neural network with an external retrieval system to produce text.
What non-parametric memory does RAG use?
In the RAG models introduced by the authors, non-parametric memory consists of an index of Wikipedia. This dense vector index is queried using a pre-trained neural retriever to supply external passages.
How does RAG compare to standard language models?
RAG models outperform parametric-only models on several open domain question answering evaluations. In language generation tasks, RAG generates language that is more factual, diverse, and specific than parametric-only sequence-to-sequence models.