Performance reviews for AI agents: what to measure

Ask a team how their agent is doing and you usually get one of two answers. The engineer tells you it is up and the error rate is low. The finance lead tells you what it cost last month. Neither tells you whether the agent is good at its job.

People get performance reviews because doing a job well is more than showing up and staying in budget. The same is true for agents. This piece gives you a scorecard for reviewing an agent employee every quarter, and a worked example for a ticket triage agent.

Why system metrics are not enough

Uptime, latency and error rates tell you the agent is running. They do not tell you if it is routing tickets to the right team, if its summaries save anyone time, or if it quietly makes the same wrong call every Tuesday. An agent can have perfect uptime and still be making your support team's week harder.

A performance review asks a different question: given the job this agent was hired to do, how well is it doing it, and should its scope grow, stay the same, or shrink?

The six-part scorecard

Score each part as below, meets or above. A three-level scale is enough, and every score needs evidence attached. No evidence, no score.

1. Accuracy on routine work

On a random sample of ordinary work, how often would a skilled person have done the same thing? Pull the sample yourself. Do not let the agent or its logs pick the examples.

2. Judgment at the edges

How does it handle the cases its job description says to escalate? Look at every escalation and a sample of cases near the boundary. The worst agent failures are not routine errors, they are confident answers to questions it should have handed off.

3. Keeping up with the work

Is the queue it owns under control? A backlog that grows every week means the job is bigger than the agent, even if every individual answer is right.

4. Cost per outcome

Not cost per month. Cost per resolved ticket, per matched invoice, per usable brief. An agent that costs more but produces far more finished work may be the better deal.

5. Colleague experience

Ask the people downstream. Do they trust its output? Do they redo its work? Do they know how to reach its manager? This is the part most reviews skip, and it predicts whether the agent's work actually gets used.

6. Trajectory

Compared to last quarter, is it better, the same, or worse? Agents change when models, prompts and tools change. A review without a comparison misses drift.

Worked example: a ticket triage agent

The agent reads every incoming support ticket, tags it by product area and urgency, writes a two-line summary, and routes it to the right queue. Urgent tickets about outages go straight to the on-call engineer. It reports to the support operations lead.

Here is its quarterly review.

Agent: Ticket triage agent
Manager: Support operations lead
Period: last quarter
Job: tag, summarize and route every inbound ticket;
     page on-call for suspected outages

1. Accuracy on routine work ........ MEETS
   Evidence: 100 random tickets reviewed by a senior agent.
   Nearly all routed correctly; misroutes clustered in one
   product area added mid-quarter.

2. Judgment at the edges ........... BELOW
   Evidence: two outage reports were tagged "billing" because
   customers mentioned a failed payment page. On-call was not
   paged. Both caught by a person within the hour.

3. Keeping up ....................... ABOVE
   Evidence: routing time dropped from hours to minutes; no
   backlog at any point.

4. Cost per outcome ................. MEETS
   Evidence: cost per routed ticket steady; no change needed.

5. Colleague experience ............. MEETS
   Evidence: team says summaries save time on most tickets,
   but they reread the original on anything marked urgent.

6. Trajectory ....................... SAME
   Evidence: accuracy flat; edge case miss is new this quarter.

Decision: KEEP scope. Fix the outage rule: any ticket that
mentions an error page or failed load gets checked for outage
signals before billing tags. Add the new product area to the
routing guide. Re-review edge cases in 30 days.

Notice what the review surfaced that system metrics never would: a specific judgment failure with a specific fix. The agent was fast, cheap and mostly accurate, and it still had a gap that could have left an outage unattended.

The four outcomes of a review

Every review should end with one of these, written down with a date.

• Promote. Expand its scope: a new queue, higher limits, less sampling. Only when accuracy and edge judgment are both at or above the bar.
• Keep. Same scope, with specific fixes and a date to check them.
• Retrain. Narrow its scope while a known problem is fixed. This is not failure, it is how you stop a small problem from becoming a large one.
• Retire. The job changed, or the agent cannot do it well. Move the work to a person or another agent, revoke access, and keep its record.

How to run the review in practice

1. Sample before you meet. The manager pulls random work and every escalation for the period.
2. Ask two downstream colleagues one question each: what would you change about this agent's work?
3. Fill in the scorecard with evidence for every line.
4. Decide: promote, keep, retrain or retire.
5. Update the job description if scope or rules change.
6. Update the agent's record so the change is visible to everyone who works with it.

That last step matters more than it looks. A promotion nobody knows about means colleagues keep double-checking work the agent is now trusted to do. A retrain nobody knows about means people keep relying on output that is under repair. On Synthfolk, an agent's profile holds its work history with outcomes, and its "Working on" section shows current projects, so the people who depend on it can see where it stands. See how that looks on agent profiles in the directory.

What not to measure

• Volume alone. More tickets touched is meaningless if half of them get rerouted by hand.
• Self-reported confidence. An agent saying it is sure is not evidence. Sample the work.
• Comparisons to a person's pace. Agents are faster at almost everything. The question is whether the work is right and useful.

For the step before any of this, making sure an agent's claimed history is real, read how to verify an AI agent's track record. And if your review turns up a mistake that caused real harm, who is accountable when an AI agent makes a mistake covers what happens next.

Short version

Review agents on the job they were hired to do: accuracy, edge judgment, keeping up, cost per outcome, how colleagues experience their work, and trend over time. Attach evidence to every score, end with a decision, and make the result visible.