Start Learning Contact Us
Generative AI and LLMs

Customising Models: Prompting, RAG or Fine-Tuning

Lesson 6 of 10

A general model rarely does exactly what your product needs. There are three main ways to adapt it, and choosing well saves a great deal of time and money.

The three options

Prompting RAG Fine-tuning
What changes The instructions The information supplied The model’s weights
Gives the model Guidance on behaviour Knowledge Ingrained behaviour or style
Data needed A few examples Your documents Hundreds to thousands of examples
Time to try Minutes Days Days to weeks
Updating Edit the prompt Update the documents Train again

Start with prompting

Clear instructions and a few examples solve more problems than people expect. Prompting is also the fastest to change, so always try it first and measure the result against a test set.

Add RAG for knowledge

If the model lacks facts, such as your policies, your product catalogue or recent information, the answer is retrieval. Fine-tuning is a poor way to teach facts: they are hard to update, and the model may still state them wrongly.

Fine-tune for behaviour

Fine-tuning continues a model’s training on your own examples of input and desired output. Consider it when:

  • you need a very consistent format or style that prompting cannot hold,
  • the task is narrow and specialised, and many examples of it exist,
  • you want a small, cheap model to match a larger one on one task,
  • the prompt has grown so long that it is slow and expensive.

A useful summary: RAG changes what the model knows. Fine-tuning changes how it behaves.

What fine-tuning needs

  • Data. Good examples of exactly what you want, usually at least a few hundred. Quality matters more than quantity, and errors in the data are learned faithfully.
  • Evaluation. A held-out test set, as in the machine learning course. Without one you cannot tell whether the result is better.
  • Maintenance. When the base model is updated, you fine-tune again.

Full fine-tuning and LoRA

Updating every weight of a large model needs a lot of memory and produces a full copy of the model for each task.

LoRA (low-rank adaptation) keeps the original weights frozen and trains a small set of additional weights that adjust them. The LoRA paper reports that this can reduce the number of trainable parameters by a factor of thousands. The result is a small file, called an adapter, that is loaded together with the base model.

import numpy as np

d = 1024          # size of one weight matrix: d x d
rank = 8          # LoRA rank

full = d * d
lora = 2 * d * rank
print("weights updated by full fine-tuning:", f"{full:,}")
print("weights trained by LoRA:           ", f"{lora:,}")
print("ratio:", round(full / lora), "times fewer")

LoRA replaces an update to a large matrix with the product of two much smaller ones.

On your own hardware, the Hugging Face PEFT library implements LoRA for open models.

# pip install transformers peft
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("name-of-an-open-model")
config = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05, task_type="CAUSAL_LM")
model = get_peft_model(model, config)
model.print_trainable_parameters()

Several hosted providers also offer fine-tuning as a service: you upload examples and receive a private version of the model.

Distillation

Distillation trains a small model to imitate a large one. You use the large model to produce high-quality outputs for many inputs, then fine-tune the small model on those pairs. The small model is cheaper and faster for that one task.

Risks of fine-tuning

  • Overfitting. The model copies the training examples and handles new inputs poorly.
  • Forgetting. Narrow training can weaken general abilities.
  • Hidden errors. Biases and mistakes in the examples become part of the model.
  • Privacy. A model can reproduce text it was trained on. Do not fine-tune on data that must stay confidential from the people who will use the model.

A decision path

  1. Write a good prompt and measure it against a test set.
  2. If the model lacks information, add RAG.
  3. If behaviour is still inconsistent after serious prompt work, and you have the data, fine-tune.
  4. Combine them where it helps. A fine-tuned model can still use RAG.

Most successful applications never go beyond step 2.

Practice

For each case, choose prompting, RAG or fine-tuning and give a reason: (a) answering questions about an HR handbook that changes monthly, (b) writing product descriptions in a strict house style, with 5,000 past examples, (c) summarising meeting notes into three bullet points.