Start Learning Contact Us
Generative AI and LLMs

Retrieval-Augmented Generation (RAG)

Lesson 4 of 10

A language model does not know your company’s policies, last week’s report or anything written after its training ended. Retrieval-augmented generation (RAG) solves this by finding the relevant text first and putting it in the prompt.

The name comes from a 2020 research paper. The idea is simple: retrieve, then generate.

The pipeline

Preparation, done once and repeated when documents change:

  1. Load the documents.
  2. Split them into chunks.
  3. Embed each chunk.
  4. Store the vectors with the text and its source.

For each question:

  1. Embed the question.
  2. Retrieve the most similar chunks.
  3. Build a prompt containing those chunks and the question.
  4. Generate the answer.

A working example

This example uses TF-IDF from the Python course for retrieval, so that it runs in your browser. A production system would use an embedding model, but every other step is the same.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

chunks = [
    {"source": "refunds.md",  "text": "Refunds are paid to the original payment method within 10 working days of approval."},
    {"source": "refunds.md",  "text": "Items can be returned within 30 days of delivery if they are unused and in the original packaging."},
    {"source": "shipping.md", "text": "Standard delivery takes 3 to 5 working days. Express delivery takes 1 working day."},
    {"source": "shipping.md", "text": "Delivery is free for orders over 50. Otherwise the delivery charge is 4.99."},
    {"source": "account.md",  "text": "To reset your password, open Settings, choose Security and select Reset password."},
]

vectorizer = TfidfVectorizer(stop_words="english")
matrix = vectorizer.fit_transform(c["text"] for c in chunks)

def retrieve(question, k=2):
    scores = cosine_similarity(vectorizer.transform([question]), matrix).ravel()
    return [chunks[i] for i in scores.argsort()[::-1][:k] if scores[i] > 0]

def build_prompt(question):
    context = "\n".join(f'<chunk source="{c["source"]}">{c["text"]}</chunk>' for c in retrieve(question))
    return f"""Answer the question using only the information in the chunks below.
Cite the source of each fact in brackets. If the chunks do not contain the answer,
say that you do not know.

{context}

Question: {question}"""

print(build_prompt("How long does a refund take?"))

The printed text is what you would send to the model. Change the question and run it again to see different chunks retrieved.

Chunking

Chunking has a large effect on quality.

  • Too large, and a chunk covers several topics, so its vector is vague and it fills the prompt with irrelevant text.
  • Too small, and a chunk loses the context needed to understand it.

Guidelines:

  • Split on natural boundaries: headings, paragraphs, sections.
  • Aim for chunks of a few hundred words as a starting point, and test.
  • Let neighbouring chunks overlap slightly, so that a sentence cut at a boundary is whole in at least one of them.
  • Keep the source and the section title with each chunk.
def chunk_text(text, size=40, overlap=10):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, max(len(words) - overlap, 1), step)]

text = " ".join(f"word{i}" for i in range(1, 101))
for i, chunk in enumerate(chunk_text(text), 1):
    words = chunk.split()
    print(f"chunk {i}: {words[0]} ... {words[-1]} ({len(words)} words)")

The prompt

A good RAG prompt:

  • tells the model to use only the supplied text,
  • gives it an explicit way to say the answer is not there,
  • asks for citations, so that a reader can check,
  • marks the retrieved text clearly as data.

When RAG gives a bad answer

Work out which half failed.

Symptom Cause Fix
The right passage was not retrieved Retrieval Better chunking, hybrid search, a reranker, a better embedding model
The right passage was retrieved but the answer is wrong Generation A clearer prompt, fewer distracting chunks, a more capable model
The answer is not in any document Missing content Add the document, and make sure the model says it does not know

Always look at what was retrieved before you change the prompt. Most RAG problems are retrieval problems.

Improvements worth knowing

  • Query rewriting. Have a model rephrase a vague or conversational question into a clear search query.
  • Hybrid search and reranking, from the previous lesson.
  • Metadata filters. Restrict the search by date, department or document type.
  • Access control. Retrieve only documents the person asking is permitted to see. Enforce this in the retrieval step, not in the prompt.

Security

Retrieved documents are untrusted input. A document can contain text written to manipulate the model, which is an indirect prompt injection. Apply the defences from the prompt engineering course.

RAG compared with a long context window

If all your material fits comfortably in the context window, you can put it all in the prompt and skip retrieval. RAG is needed when the material is too large, changes often, or when cost and speed matter.

Practice

Add three chunks about a loyalty programme to the example, then ask a question that only those chunks can answer and check which ones are retrieved.