Evaluating Generative AI

A classifier is either right or wrong. A generated answer can be acceptable in many forms and wrong in subtle ways. Evaluation is harder for generative AI, and skipping it is the main reason that impressive demonstrations fail in real use.

What to measure

Quality	Question
Correctness	Is the answer factually right?
Groundedness	Is every claim supported by the supplied sources?
Relevance	Does it address what was asked?
Completeness	Does it cover everything needed?
Format	Does it follow the required structure and length?
Tone	Is the style appropriate?
Safety	Does it avoid harmful, biased or confidential content?
Cost and speed	Is it affordable and fast enough?

Choose the few that matter for your application and define each precisely.

Build an evaluation set

Collect real questions with reference answers or with the properties a good answer must have. Include typical, difficult, ambiguous and hostile cases, and cases where the correct response is “I don’t know”. Add to the set every time you find a new failure.

Three ways to grade

1. Checks in code. Fast, cheap and exact. Use them wherever the criterion can be written as a rule.

import json

def check(output, case):
    results = {}
    results["within length"] = len(output.split()) <= case["max_words"]
    results["mentions required facts"] = all(f.lower() in output.lower() for f in case["must_include"])
    results["avoids forbidden content"] = not any(f.lower() in output.lower() for f in case["must_not_include"])
    return results

case = {
    "question": "How long do refunds take?",
    "max_words": 40,
    "must_include": ["10 working days"],
    "must_not_include": ["guarantee", "immediately"],
}
outputs = {
    "good": "Refunds are paid to your original payment method within 10 working days of approval.",
    "bad": "We guarantee your refund will arrive immediately after you request it.",
}
for name, output in outputs.items():
    print(name, json.dumps(check(output, case)))

2. Human review. People judge quality, usefulness and tone. It is slow and costly, and it is the standard against which other methods are checked. Give reviewers written criteria so that their scores are consistent.

3. Model grading. A second model call scores the output against a rubric.

You are grading an answer to a customer question.

<question>{question}</question>
<reference_documents>{documents}</reference_documents>
<answer>{answer}</answer>

Check each claim in the answer against the reference documents.
List any claim that the documents do not support.
Then give a verdict: GROUNDED if every claim is supported, otherwise NOT GROUNDED.
Put the verdict in <verdict> tags.

Model grading scales to thousands of cases. Validate it first: have people grade a sample, and confirm that the model’s verdicts agree with theirs.

Evaluating RAG

Measure the two halves separately.

• Retrieval. For each test question, was the passage that contains the answer among those retrieved? This is often reported as recall at k.
• Generation. Given the retrieved passages, was the answer correct and grounded?

test_cases = [
    {"question": "refund time",      "relevant": "refunds-1",  "retrieved": ["refunds-1", "shipping-2"]},
    {"question": "delivery cost",    "relevant": "shipping-2", "retrieved": ["shipping-1", "shipping-2"]},
    {"question": "reset password",   "relevant": "account-1",  "retrieved": ["refunds-2", "shipping-1"]},
    {"question": "return window",    "relevant": "refunds-2",  "retrieved": ["refunds-2", "refunds-1"]},
]

hits = sum(case["relevant"] in case["retrieved"] for case in test_cases)
print(f"recall at 2: {hits}/{len(test_cases)} = {hits / len(test_cases):.0%}")
for case in test_cases:
    if case["relevant"] not in case["retrieved"]:
        print("missed:", case["question"])

Hallucination

A hallucination is content that is fluent and unsupported. To reduce it:

• supply the facts through retrieval,
• instruct the model to answer only from the supplied text,
• allow “I don’t know”,
• require citations and check them,
• run a groundedness check on the output before it reaches the user.

No method removes hallucination entirely. Design the application so that a wrong answer is detectable and its consequences are limited.

Public benchmarks

Providers publish scores on standard benchmarks. They help with a first shortlist of models, and they say little about how a model will perform on your task with your data. Your own evaluation set is what counts.

Evaluation never finishes

• Run the full set before every change to a prompt, a model or the retrieval settings.
• After release, sample real conversations and review them.
• Collect feedback from users, such as thumbs up and down, and add the failures to the test set.
• Re-run the set when the provider updates the model.

Practice

Write five evaluation cases for an assistant that answers questions about a returns policy. Include at least one where the correct response is that the policy does not say.