Success Criteria for AI Agents: Define Done Before You Run

Most AI agent failures are not spectacular. The agent does not crash, does not leak data, does not send an unhinged email. It simply does something adjacent to the job you wanted, produces a plausible-looking summary, and moves on. You read the output, feel vaguely unsatisfied, and cannot articulate why. The reason is almost always the same: nobody defined what "done" meant before the run started.

Success criteria fix this. They are the part of an agent's configuration that turns "run and hope" into ai agent evaluation you can actually trust: a short list of measurable conditions the run report checks itself against at the end of every run. This guide covers what good criteria look like, how they work mechanically in Skopx, concrete examples for common agent use cases, and the honest limits of the approach.

Why "done" needs a definition before the first run

When a human takes on a recurring task, the definition of done lives in their head and gets refined through feedback. When an autonomous agent takes on the same task, that implicit standard has to become explicit, because there is no shared context to fall back on. The agent will interpret "triage my inbox" in whatever way its instructions and the current state of the inbox suggest, and without a stated target, any output can look like success.

This is the core problem with evaluating agents by vibes. A run that ends with a confident, well-formatted report is not necessarily a run that did the job. Three failure modes show up constantly:

• Partial completion presented as full completion. The agent processed 12 of 40 emails, hit an ambiguity, and wrote a report about the 12. Nothing in the report says "28 were skipped" unless you made coverage a criterion.
• The right activity against the wrong target. The agent monitored competitors, but only the two it found easily, not the five you care about. Activity happened; the job did not.
• Silent scope drift over time. Run one does the job. Run fifteen, after upstream data changed shape, does half the job. Without criteria evaluated every run, drift is invisible until something breaks downstream.

Success criteria do not prevent these failures outright. What they do is make each one visible in the run report, on the run where it first happens, instead of weeks later.

How success criteria work in Skopx

In Skopx, success criteria are a first-class part of an agent's anatomy, alongside its plain-language instructions, its trigger, its per-integration grants, and its budgets. You define them the same way you define everything else in Create Agent: by describing them in chat. No code, no canvas. You say something like "the run succeeds only if every unread email older than one hour has been labeled, and the report lists any email it could not classify," and the chat assembles that into the agent's configuration.

Mechanically, the criteria matter at the end of every run. Each run in Skopx ends with a markdown report, rendered as a document, and that report evaluates the run against the success criteria you defined. This is the crucial design point: evaluation is not a separate dashboard you have to remember to check, and it is not something you eyeball across a step timeline. The report itself answers "did this run meet the bar?" and shows its reasoning, with the full step timeline, duration, and token count sitting underneath it if you want to verify. For a deeper look at what those reports contain and how to read them, see what goes into an AI agent's run report.

Because run history is append-only, every run's self-evaluation is preserved. You get a longitudinal record of how often the agent meets its criteria, which is the raw material for deciding whether to trust it with more, tighten its leash, or rewrite its instructions.

The anatomy of a good criterion

A useful success criterion has three properties. Miss any one of them and the criterion degrades into decoration.

It is observable from within the run. The agent can only evaluate what it can see: the data it fetched, the actions it took or parked for approval, the tool responses it received. "Customer satisfaction improved" is not observable from a run. "Every ticket tagged urgent received a drafted reply" is.

It is measurable, ideally countable. "Summarize the important changes" invites judgment calls. "List every competitor page that changed since the last run, with a one-line note per change, or explicitly state that none changed" can be checked mechanically. Numbers, coverage statements, and explicit none-found clauses are your friends.

It defines failure, not just success. The most valuable criteria are the ones that force the agent to admit shortfalls. "The report must name any account the agent could not access and any record it skipped" turns silent partial completion into a stated, reviewable fact. Candid failure reporting is worth more than optimistic success reporting.

A practical test: could a colleague who never saw the agent read the report and the criteria, and decide in under a minute whether the run did its job? If not, the criteria are too soft.

There is one more property worth stating because it is counterintuitive: good criteria are few. Three to five sharp criteria beat twelve vague ones, because every criterion is something the report must evaluate and something you must read. If you find yourself writing criterion number eight, you probably have two agents' worth of job in one agent.

Vague versus measurable: a side-by-side

The fastest way to internalize the difference is to see the same intent expressed both ways.

Notice the pattern in the right-hand column: coverage is quantified ("every", "all 5"), the negative case is forced into the open ("zero changes is stated explicitly"), and the unit of evidence is named (invoice ID, rubric line, missing field). None of these criteria require the agent to be smarter. They require it to be accountable.

Success criteria by use case

Criteria are only as good as their fit to the job. Here is how they play out across common agent patterns, all of which you can build in Skopx by describing them in chat.

Monitoring agents (competitor pages, price changes, SEO positions, uptime logs). The core criterion is coverage plus explicit deltas: every tracked target was checked, every change from the stored baseline is reported, and no-change is stated rather than implied by silence. Because Skopx agents have memory that persists between runs, the baseline lives in the agent itself: the first run establishes it, and second runs produce delta reports against it, which are typically cheaper too. How that persistence works is covered in how AI agent memory works between runs.

Triage agents (inbox, support queues, bug reports). The core criterion is exhaustiveness with an honest remainder: every item in the queue got exactly one disposition, and items the agent could not confidently classify are listed rather than force-fitted. A worked example of this pattern is in building an inbox triage agent.

Drafting agents (email replies, renewal reminders, review responses). Here the criteria split into coverage and restraint: a draft exists for every item meeting the trigger condition, and nothing was sent, because sending is parked behind approval. In Skopx, write-shaped actions park as pending approvals showing the exact call and arguments, so "all outbound actions are parked, none executed" is a criterion the report can verify directly against the run's own record.

Analysis agents (KPI digests, churn signals, expense audits, data quality checks). The core criteria are completeness of the metric set and honesty about gaps: every defined metric is present, every failed query is surfaced as unavailable rather than silently dropped, and any figure in the report traces to a query the step timeline actually shows. Skopx agents query connected data sources with read-only SQL and bound parameters, so the evidence trail is inspectable.

Research agents (market scans, meeting prep, recruiting screens). These are the hardest to pin down, because "good research" resists counting. The workable move is to define criteria around structure and sourcing rather than insight: every claim in the report links to a fetched source, every named entity on the input list was investigated, and open questions are listed as open rather than papered over.

Criteria, budgets, and approvals work as one system

Success criteria do not operate in isolation. In Skopx they sit inside a guardrail system, and the interactions matter.

Budgets bound the attempt; criteria judge the result. Every agent carries budgets: tokens per run, tokens per day, a max step count, and a minute cap. A run that hits its budget stops, and three budget failures auto-pause the agent entirely. This creates a healthy tension you should design for deliberately. If your criteria demand "check all 200 accounts" but your step budget supports 60, every run will fail its criteria honestly, which is the system working: it is telling you to either raise the budget or narrow the job. The interplay is explored further in how token budgets keep agents bounded.

Approvals shape what "done" can mean. For an agent whose grants require asking first, a successful run often ends with work parked, not work executed. Write your criteria accordingly: "a correct draft is parked for every qualifying item" rather than "every item was handled." Approving a parked action executes exactly that call once; rejecting it executes nothing. The criterion evaluates the parking, and you evaluate the parked call.

Memory lets criteria get stricter over time. First-run criteria and steady-state criteria are different animals. A first run establishing a baseline cannot be judged on deltas. A reasonable pattern: run one succeeds if the baseline is complete; every subsequent run succeeds only if the delta report covers all tracked items. Because the agent's memory persists cursors and baselines between runs, this distinction is expressible in plain language in the instructions.

Writing criteria the report can actually check

A few drafting habits make the difference between criteria that get evaluated and criteria that get restated.

Quantify coverage or it does not exist. "Process the important emails" gives the agent license to define "important" downward under pressure. "Process every unread email; if the count exceeds what the budget allows, process oldest first and report the exact number left" gives it no such license.

Force the null result into words. The single highest-value clause you can add to almost any monitoring or triage criterion is "if nothing qualified, say so explicitly." It converts the ambiguous silence of an empty report section into a positive, checkable statement.

Name the evidence unit. "The report cites its sources" is soft. "Every flagged deal appears with its CRM record ID" is checkable against the step timeline, where every raw tool result is expandable.

Separate criteria from instructions, but keep them consistent. Instructions say how to do the job; criteria say how to know it was done. They should agree. If your instructions say "focus on enterprise accounts" and your criteria say "every account was checked," the agent is set up to fail one or the other. When you edit one, reread the other. The craft of the instructions side is its own topic, covered in writing effective AI agent instructions.

Prefer criteria a skeptic would accept. Imagine the report being read by someone who assumes the agent cut corners. Which sentences would convince them otherwise? Those sentences, generalized, are your criteria.

Iterating on criteria with run history

Your first draft of success criteria will be wrong in at least one direction: too loose somewhere, too strict somewhere else. This is normal, and the append-only run history is the tool for fixing it.

The loop looks like this. Run the agent on its real trigger for a week or two. For each run, compare the report's self-evaluation against your own reading of the step timeline. You are looking for two kinds of mismatch:

• False passes: the report says criteria were met, but you can see from the timeline that the job fell short. This means a criterion is missing or unmeasurable. Tighten it: add the coverage number, the evidence unit, the null-result clause.
• False failures: the report says criteria were not met, but the run did what you actually wanted. This usually means a criterion encodes an ideal rather than the real job. Loosen it honestly rather than letting perpetual failure train you to ignore the evaluation.

Because instructions in Skopx are editable and versioned, you can tighten criteria without losing the history of what you changed and when. And because each run's report is preserved, you can check whether an edit actually moved the pass rate. If a run fails and you cannot tell why from the report alone, drop into the step timeline and expand the raw results; the workflow for that is laid out in debugging AI agent runs. It is also worth doing your first several iterations with manual triggers and drafts-only grants before letting a schedule take over, an approach detailed in testing AI agents safely.

Expect the criteria to stabilize. A well-fitted agent settles into passing its criteria on nearly every run, and at that point the evaluation section of the report becomes a cheap, continuous regression test: the day it starts failing again, something in the environment changed, and you will know that day rather than a month later.

Where success criteria fall short

Candor requires saying what this mechanism cannot do.

Self-evaluation is not independent evaluation. The run report is written by the same model that did the work, judging its own output against your criteria. Sharp, countable criteria constrain that judgment substantially, but they do not eliminate it. An agent can misjudge whether a summary is "accurate" far more easily than it can misjudge whether "all 5 pages were fetched." This is exactly why the step timeline with expandable raw results exists: the evaluation tells you where to look, and the evidence lets you verify. Trust the countable criteria more; spot-check the judgment-based ones.

Criteria cannot measure quality you cannot articulate. If you cannot state what a good competitive brief looks like, no criterion will conjure that standard. The honest move is to keep such agents on manual triggers and human review until repeated reading teaches you what you actually want, then encode it.

Passing criteria is not business impact. An invoice-chasing agent can park a perfect draft for every overdue invoice and still not move cash collection if the drafts are never approved or the emails never work. Criteria measure the agent's execution of its job as defined. Whether the job as defined is worth doing is a question the criteria cannot answer, and pretending otherwise is how teams end up with agents that pass every run and matter to no one.

Some jobs should not be delegated at all, regardless of criteria. High-stakes, low-volume, judgment-heavy decisions do not become agent-safe because you wrote a rubric. Success criteria make delegation measurable; they do not make every delegation wise.

None of these limits argue against writing criteria. They argue for treating criteria as one layer in a system that also includes grants, budgets, approvals, and your own reading of reports.

FAQ

How many success criteria should an agent have?

Three to five is the sweet spot for most agents. Fewer than three usually means coverage or honesty is going unmeasured. More than six usually means either redundancy or an agent doing two jobs that should be split into two agents, each with its own tight criteria. Every criterion is a promise the report must evaluate on every run, so each one should earn its place.

What happens in Skopx when a run does not meet its criteria?

The run report states which criteria were not met and why, and the run is preserved in the append-only history like any other. A criteria failure by itself does not pause the agent; auto-pause is tied to budget failures (three of them) rather than evaluation results. This is deliberate: a failed evaluation is information for you to act on, and the report is where you act on it. If failures repeat, edit the instructions or the criteria, or pause the agent yourself, which also acts as a kill switch for queued runs.

Can success criteria stop an agent from taking a bad action?

No, and it is important to be clear about this. Criteria are evaluated in the run report; they judge outcomes, they do not gate actions. Action gating is the job of grants and approvals: an integration set to ask first parks every write-shaped call as a pending approval showing the exact call and arguments, and nothing executes until you approve it. Use grants and approvals to control what the agent may do, budgets to bound how much it can attempt, and success criteria to measure whether what it did was the job.

Should criteria differ between the first run and later runs?

Usually yes, for any agent that builds a baseline. A first run cannot be judged on deltas it has no baseline to compute, so a sensible pattern is: run one succeeds if the baseline is completely established, and later runs succeed only if every tracked item is covered in the delta report. Because agent memory persists cursors and baselines between runs, you can express this two-phase standard directly in plain language when you create the agent.

Do different models need different success criteria?

The criteria should stay the same; they describe the job, not the worker. What changes across models is the pass rate. Skopx lets you pick the model per agent across providers including Claude, GPT, Gemini, and Kimi, and stable criteria are precisely what make model comparison meaningful: run the same agent, same criteria, different model, and read the evaluations side by side. If you tune criteria per model, you lose the ability to compare anything.