How Language Models Read Your Prompt

Prompt engineering is the practice of writing instructions that get accurate, consistent results from a large language model (LLM). It is easier to do well once you know how a model handles your text.

A model predicts the next token

A language model is trained to predict what comes next in a piece of text. Given your prompt, it produces one small unit of text, adds it to the input, and repeats until the answer is complete.

The model has no goal of its own and no view of what you meant. It has only the text in front of it. This is why wording matters so much: the prompt is the model’s entire understanding of the task.

Tokens

Models work on tokens, not words. A token is a common piece of text: a short word, part of a longer word or a punctuation mark. As a rough guide for English, a token is about three quarters of a word.

Tokens matter for three reasons:

• Limits on input and output are measured in tokens.
• You are usually charged per token.
• Tasks that depend on individual letters, such as counting the letters in a word, can go wrong because the model does not see letters.

The context window

The context window is everything the model can see at once: instructions, documents you supplied, the conversation so far and the answer it is writing. Anything outside it does not exist for the model.

No memory between requests

Unless the application adds it, a model remembers nothing from one request to the next. A chat feels continuous because the application sends the whole conversation again each time. If you start a new conversation, you must supply the context again.

A training cut-off

A model learned from data collected up to a certain date. It does not know about later events unless you put the information in the prompt or give it a search tool.

Sampling and temperature

At each step the model assigns a probability to every possible next token and one is chosen. Temperature controls how the choice is made.

import numpy as np

tokens = ["Paris", "London", "the", "a", "Lyon"]
scores = np.array([6.0, 2.5, 2.0, 1.5, 3.0])   # the model's raw scores

def probabilities(scores, temperature):
    scaled = scores / temperature
    e = np.exp(scaled - scaled.max())
    return e / e.sum()

for t in (0.2, 1.0, 2.0):
    p = probabilities(scores, t)
    print(f"temperature {t}:", dict(zip(tokens, p.round(3))))

A low temperature concentrates the probability on the most likely token, giving focused and repeatable answers. A high temperature spreads it out, giving more varied answers. Use low values for extraction and analysis, and higher values for brainstorming.

Even at a low temperature, the same prompt can produce slightly different answers. Do not assume exact repeatability.

Why models make things up

A model produces text that is plausible given its training, which is not the same as text that is true. When it lacks the information, it may still produce a confident, fluent answer. This is called hallucination.

You reduce it by:

• giving the model the facts it needs in the prompt,
• telling it that saying “I don’t know” is acceptable,
• asking it to quote the source text that supports each claim,
• checking anything that matters.

What this means for prompting

Think of the model as a very capable new colleague who knows a great deal in general but nothing about your situation, your organisation or what you discussed yesterday. A prompt that would be clear to that person is usually a good prompt.

Practice

Run the temperature example with a temperature of 0.05 and of 5. Describe the difference in one sentence each.