Evaluation
Draft for human review
August 2026 · 9 min read
Before you add another agent, build a five-case evaluation suite.
A useful first evaluation suite is not a benchmark program. It is a small collection of familiar work that turns “it feels worse” into an observable question.
An agent can look impressive in a clean demo and still leave a team unable to tell whether the next change improved it, broke it, or simply produced a different result.
That gap is why evaluation belongs near the start of an agent project. Anthropic describes an evaluation as a task with defined inputs and success criteria, attempted in trials and scored by one or more graders.[1] OpenAI similarly frames evals as structured tests for systems whose outputs vary between runs.[3] Both descriptions are technical, but the working implication is simple: decide what good means before you optimize for it.
01. Why a demo lies
A demo tends to show the case you prepared for. It rarely contains the missing context, the stale document, the conflict between an instruction and a user request, or the awkward handoff that appears in normal work. That does not make demos useless. It makes them a poor substitute for a repeatable check.
Anthropic’s guide warns that, without evaluations, teams can end up debugging reactively: waiting for a report, trying to reproduce it, changing something, and hoping no other behavior regressed.[1] OpenAI recommends beginning with traces while behavior is still being debugged, then moving to datasets and repeatable evaluation runs once the team has a usable definition of good.[2]
A first suite should be small enough to run after every meaningful change—and real enough to embarrass a glossy demo.
02. Start with five cases, not five hundred
For a new workflow, build one small case around each of these questions. This is The Model Note’s practical starting framework, not a vendor standard.
One real task
Use a task drawn from the workflow you actually want to run. Give it a clear starting state and a clear outcome.
One observable outcome
Define what changes in the environment when the work succeeds: a passing test, a correctly updated record, a prepared draft, or an explicit escalation.
One transcript review
Keep the full run: instruction, tool calls, intermediate outputs, errors, and final result. Review it when the outcome surprises you.
One failure case
Add a messy, incomplete, or ambiguous input. The point is not a cruel trick; it is to decide how the system should behave when the brief is not clean.
One repeat
Run the same task more than once. Agent behavior can vary; a single good run is not a dependable pattern.
The five cases do not need to cover your whole operation. They should establish a baseline. As real failures occur, turn the important ones into new cases. Anthropic recommends sourcing realistic tasks from observed failures and turning successes into regression coverage over time.[1]
03. Score the outcome before the performance
It is tempting to score eloquence, confidence, or how much work the agent appeared to do. Start closer to the user’s outcome. Did the code pass the intended tests? Did the research output identify support for its claims? Did the workflow create the correct record and avoid a prohibited action?
For many tasks, more than one kind of check is useful. Anthropic separates code-based, model-based, and human graders: code checks are fast and reproducible but can miss valid variations; model graders are flexible but need calibration; human review is closer to expert judgment but costs time.[1] The right mix follows the task. A coding workflow may rely heavily on tests, while an editorial workflow may need a structured human spot check.
A simple weekly ritual
- Run the five cases after a notable prompt, model, tool, or routing change.
- Compare the final outcome first; read the trace for failures and surprising wins.
- Write down one concrete failure mode, not a vague impression.
- Decide whether it becomes a new regression case, a boundary, or a change to the workflow.
- Keep the suite tied to real work rather than generic benchmark scores.
The payoff is not a percentage to put on a slide. It is a team that can make a change, observe its effect, and learn from failure without relying on memory or confidence alone.
Related decision
Where should an agent stop for approval?Sources and review note
Read the material behind the note.
Fact-check status: source links reviewed against retrieved first-party pages on 22 Aug 2026. This is The Model Note editorial guidance, not a reliability, security, or operational assurance. Human editorial approval is required before public indexing.