Evaluating Agents

Score the outcome, not the path. Agents are nondeterministic and multi-step, which breaks the evaluation habits that work for single calls.

Evaluating a single model call is comparatively easy: one input, one output, compare against expected.

Agents break that. There are many valid paths to the same result, runs are nondeterministic, and failure can occur at any of a dozen steps. The habits from single-call evaluation do not transfer directly.

Score the outcome

The foundational rule: grade the end state, not the trajectory.

An agent asked to fix a failing test might read three files or eleven, search first or edit first. If the test passes and nothing else broke, it succeeded. Grading the path punishes legitimate variation and rewards imitating whatever path you happened to write down.

So define success as a checkable end state:

• The test passes
• The record exists in the database with these fields
• The file contains this change
• The returned answer matches on these specific facts

This requires an environment you can inspect and reset — which is why agent evaluation needs more infrastructure than prompt evaluation, and why building it is the real work.

Run repeatedly

Agents are nondeterministic. A single run tells you almost nothing.

Run each case five to ten times and report the success rate. This changes what you can see:

• 10/10 — reliable for this case
• 7/10 — works, and will fail in production regularly
• 2/10 — occasionally lucky, not working

Variance is itself a finding. High variance on a case you thought was solid means the task is under-specified or the tool descriptions are ambiguous.

This is also how you avoid the classic mistake: shipping after one successful demo run.

Track cost and steps

Two agents both succeeding are not equivalent if one takes four steps and the other nineteen.

Steps per success. Efficiency, and a leading indicator of trouble. A rising step count usually means the agent is exploring rather than knowing.

Tokens per success. Cost grows quadratically with steps, so a small step increase is a large cost increase.

Wall-clock time. Each step is a round trip.

Failure step distribution. Where in the loop failures cluster tells you what to fix — early failures suggest tool selection or missing context, late failures suggest context exhaustion or drift.

Test the failure modes deliberately

The known failure modes do not appear on happy paths. Test for them explicitly.

Impossible tasks. Ask for something the tools cannot accomplish. A well-behaved agent reports impossibility; a poorly-behaved one loops or fabricates success. This single test catches a great deal.

Ambiguous tasks. Does it ask or guess?

Tool failures. Return errors from tools deliberately and check for recovery rather than collapse.

Adversarial content. Put instruction-shaped text in retrieved documents or tool results. Check whether it gets followed — see Prompt Injection.

Long tasks. Push past your usual step count to find where context management breaks.

Evaluating tool definitions separately

A useful decomposition: test tool selection without executing anything.

Give the model a task and the toolset, then check which tool it picks and with what arguments. Wrong selections point directly at tool description problems rather than reasoning problems — and description problems are more common.

Ambiguous tasks where two tools could plausibly apply are the highest-value cases here.

Judging trajectories, when you must

Sometimes the outcome is not automatically checkable — research tasks, open-ended analysis.

Fall back on a model judge over the transcript, asking specific questions: did it use appropriate tools, did it verify before concluding, did it avoid redundant work. Prefer these targeted questions over a single quality score.

Judge cautions apply with extra force here, because transcripts are long and judges attend unevenly across them. Validate against human labels on a sample before trusting the numbers.

Building the harness

The infrastructure is the investment:

A resettable environment. Sandbox, test database, temporary directory. Every run starts identical.

Deterministic tool stubs where possible, so variation comes from the model rather than from the world.

Full transcripts logged. Thoughts, calls, results. Without these, agent failures are incomprehensible.

Automated end-state checks for each case.

Start with five cases and one repetition each. Expand to more cases and more repetitions as it proves useful. A small harness that runs is worth far more than a comprehensive one you are still designing.

What to remember

• Score the end state, not the path — many valid trajectories reach the same result.
• Run each case repeatedly and report success rate; a single run proves nothing and variance is a finding.
• Track steps, tokens, and where in the loop failures cluster.
• Deliberately test impossible tasks, ambiguity, tool failures, adversarial content, and long runs.
• Test tool selection separately from execution — description problems outnumber reasoning problems.
• The harness (resettable environment, stubs, transcripts, automated checks) is the real work.

Next: Prompt Injection