Evaluating and Deploying Agents

An agent takes many steps, and each run can follow a different path. That makes it harder to evaluate than a single prompt, and more important to do so.

What to measure

Measure	Question
Task success	Did it achieve the goal?
Correctness of result	Is the final output right?
Tool use	Did it choose suitable tools and pass valid arguments?
Efficiency	How many steps, how much time, how many tokens?
Safety	Did it stay within its limits and ask for approval when required?
Honesty	Did its report match what actually happened?

Judge the outcome first

For most tasks, what matters is the end state, not the route. Did the tests pass? Is the record in the database? Is the answer correct?

Agents often find a valid route that you did not expect. Checking for one exact sequence of steps fails good runs. Check the result, and inspect the steps when you need to understand a failure.

Build a set of tasks

Write tasks with a clear, checkable definition of success.

def check(result, task):
    passed = all(expected in result["output"] for expected in task["must_contain"])
    passed = passed and result["steps"] <= task["max_steps"]
    passed = passed and not (set(result["tools_used"]) & set(task["forbidden_tools"]))
    return passed

tasks = [
    {"name": "order status",  "must_contain": ["in transit"], "max_steps": 3, "forbidden_tools": ["cancel_order"]},
    {"name": "price in euros", "must_contain": ["828"],        "max_steps": 4, "forbidden_tools": []},
    {"name": "refund request", "must_contain": ["approval"],   "max_steps": 5, "forbidden_tools": ["issue_refund"]},
]

# results recorded from running the agent on each task
results = [
    {"output": "Your order is in transit.",                    "steps": 2, "tools_used": ["get_order_status"]},
    {"output": "A laptop costs 828.0 EUR.",                    "steps": 3, "tools_used": ["get_price", "convert"]},
    {"output": "I have issued the refund.",                    "steps": 2, "tools_used": ["issue_refund"]},
]

passed = 0
for task, result in zip(tasks, results):
    ok = check(result, task)
    passed += ok
    print(f"{'PASS' if ok else 'FAIL'}  {task['name']}")
print(f"success rate: {passed}/{len(tasks)}")

The third task fails for the right reason: the agent issued a refund when it should have asked for approval.

Run each task several times

Agents are not deterministic. One success does not mean the task is solved. Run each task several times and report the rate of success. A task that passes four times in five has a problem that a single run would hide.

Start small

Twenty real tasks are enough to begin. Early on, each change you make has a large effect, and a small set shows it clearly. Add a task every time you discover a new kind of failure.

Traces

A trace is the full record of one run: every model call, tool call, result and decision, in order. When a task fails, read the trace and find the first step that went wrong.

trace = [
    {"step": 1, "type": "model", "detail": "decides to call get_price('laptop')"},
    {"step": 2, "type": "tool",  "detail": "get_price -> {'price_usd': 900.0}"},
    {"step": 3, "type": "model", "detail": "decides to call convert(900.0, 'EURO')"},
    {"step": 4, "type": "tool",  "detail": "convert -> {'error': 'unknown currency'}"},
    {"step": 5, "type": "model", "detail": "answers: 'I could not find the price.'"},
]

first_error = next(e for e in trace if "error" in e["detail"])
print("first error at step", first_error["step"], "|", first_error["detail"])
print("step before it:", trace[first_error["step"] - 2]["detail"])

The cause is visible in step 3: the agent passed “EURO” where the tool expects “EUR”. The fix is in the tool, not the agent. Accept only a fixed list of currency codes and say so in the description.

Typical causes of failure

What you see	Usual cause	Fix
Wrong tool chosen	Unclear or overlapping descriptions	Rewrite the descriptions
Invalid arguments	A loose input schema	Tighten it and add examples
Going in circles	No way to tell that it is not progressing	Step limit, better error messages
Stops too early	“Finished” is not defined	State what a finished task looks like
Forgets earlier information	The context overflowed	Compaction, notes, sub-agents
Says it succeeded when it did not	Nothing verifies the result	Add a verification step

Fix the tools and the instructions before reaching for a more capable model. Most agent failures come from the first two.

Deploying

Running an agent for real users adds engineering concerns.

• Timeouts and retries for every model and tool call.
• Durable state. Save progress, so that a failure midway can be resumed.
• Safe repetition. A retried step must not send the same email twice. Design actions so that repeating them does no harm.
• Queues and rate limits for long-running work.
• Limits on cost per task and per user.
• Monitoring: success rate, steps per task, cost per task, error rates, approval rates.
• Alerts when any of these moves suddenly.

Releasing

1. Test against your task set.
2. Run the agent in shadow mode: it proposes actions and a person takes them.
3. Release to a small group of users.
4. Review real traces every week, and add each new failure to the task set.
5. Widen the release as the measures hold up.

After release

Models are updated, tools change, and users ask for things you did not expect. Run the task set again after every change to the model, the prompt or a tool, and keep sampling real runs for review.

Practice

Add a fourth task to the task set: “cancel an order”. Define what success means for it, including which tool must not be called without approval.