AI Agent Performance Reviews

Why review AI workers at all?

Three reasons:

1. Cost-vs-value drift. A model upgrade costs 30% more. Was the quality bump worth it? Without metrics, you can't tell. The agent that was great in March may be over-spending in June because the model provider's pricing changed or the prompt grew without you noticing.
2. Model A/B. Claude Sonnet 4.6 vs Opus 4.7 vs GPT-5 vs Gemini 2.5 — they're priced differently and good at different things. Run them on the same task and let data pick. Don't pick by Twitter excitement.
3. Retirement decision. Some agents stop being worth their cost. Some get superseded by better harnesses. Some were the right answer in v1 and the wrong answer in v3 of your product. Reviewing surfaces this; not reviewing means dead agents linger and burn budget.

A fourth reason that grew in 2026 as fleets scaled past 20 agents: budget governance. CFOs and finance leaders started asking "show me the AI workforce monthly review" the way they ask for any other line-item review. The companies that have a structured review framework breeze through these conversations. The companies that don't get budget cuts they could have argued against.

The four metrics that matter

You'll see operators track 12 metrics. Most don't matter. Four do:

1. Success rate

Of N runs, how many completed without human intervention or downstream rejection? Healthy: 90-98% for well-defined tasks. Below 85%: the CLAUDE.md needs work. Above 99%: suspicious — verify the agent isn't pushing through ambiguity it should escalate.

How it's calculated: the platform's audit log marks each run as completed, escalated, or failed. Success rate = completed / total. Escalations and failures both count against success.

Common pitfall: defining success too loosely. "The agent posted a Slack message at the end" isn't success — the message could be reporting a failure. Define success as completed AND downstream-validated (the invoice posted to NetSuite without rejection; the test suite passed; the customer responded positively).

2. Cost-per-task

Compute (sandbox time) + model API cost, per completed task. Calculated automatically by the platform; surfaced in the dashboard. Compare to:

• The human alternative ("our paralegal does this for $15/task at full load")
• Other agents on the same workflow (cohort comparison)
• The same agent at last review (drift check)

How it's calculated: sum of sandbox-runtime cost (per second of compute) + model API cost (input tokens × per-token + output tokens × per-token), divided by completed tasks. Platform exposes this per agent and rolls up by fleet.

Common pitfall: mixing up "cost-per-task-completed" with "cost-per-task-attempted". A flaky agent with 70% success rate and $0.30/attempt is actually $0.43 per completed task. Track the right one.

3. Time-to-completion

Wall-clock from event-arrived to result-posted. Useful because:

• It's the operator-felt latency. "I forwarded the invoice at 9am, got the result at 9:08" beats "I forwarded the invoice yesterday, got the result this morning."
• Sudden jumps signal a problem (slow downstream system, agent got stuck).
• Trend down over time = the agent is getting better at the task. Trend up = drift; investigate.

How it's calculated: wall-clock between the trigger event and the final post / write. Includes wake time (if sandbox was idle), processing time, downstream calls, and result-posting time. Platform exposes p50, p95, p99 — track all three.

Common pitfall: tracking only p50. p95 catches outliers that p50 hides. An agent at p50 of 14 seconds and p99 of 8 minutes has a tail problem; the p50 alone wouldn't show it.

4. Escalation rate

% of runs the agent stopped on and asked for human input. Healthy: 1-5%. Zero is suspicious (the agent isn't asking when it should). 10%+ is too noisy (CLAUDE.md needs more guidance, or the task isn't agent-shaped at all).

How it's calculated: count of runs that ended with an explicit "escalated" status (vs "completed" or "failed"). Different from failures: an escalation is the agent doing the right thing (knowing it doesn't know) rather than the wrong thing (pushing through and getting it wrong).

Common pitfall: conflating escalation rate with failure rate. An agent that escalates 5% of runs and completes 95% has a 95% success rate AND a 5% escalation rate — both are healthy. The 5% escalations don't reduce the success rate.

Reading the metrics: what's signal vs noise

Not every metric movement is meaningful. Calibrate your sensitivity:

Building the review dashboard

The platform's audit log feeds the metrics view automatically. The dashboard shows, per agent:

• Success / escalation / failure counts (rolling 7-day, 30-day, 90-day)
• Average and p95 cost-per-task
• p50 / p95 / p99 time-to-completion
• Top 5 failure categories (taken from the failure taxonomy in When AI Agents Fail)
• Top 5 escalation reasons
• Trend lines for each metric over the last 30 days, 90 days
• Cohort comparison if multiple agents in the same role
• Cost vs budget cap (% used)

For larger fleets, sortable by any column — the agents that need attention bubble to the top. The "needs attention" view shows agents with metric anomalies (success down, cost up, escalation rate at zero, etc.).

Three dashboard views you'll use

A/B testing models on the same task

The platform's model layer lets you swap which underlying model an agent uses without changing the CLAUDE.md or tools. Run the same task on two configurations and compare:

The right answer is rarely "the best model." It's "the cheapest model that hits your quality bar." For most knowledge work, that's the mid-tier (Sonnet, GPT-5 standard, Gemini Pro). Reach for the top tier (Opus, GPT-5 Reasoning) only for the high-stakes 5%.

Tiered routing: model per task class

For agents that do triage-then-action, you can tier the work: use Haiku for the first pass (cheap, fast), promote to Sonnet only for items that pass triage, escalate to Opus only for items the Sonnet tier flagged. The cost math compounds:

def route_task(task):
    # Tier 1: quick triage with Haiku
    triage = call_model("claude-haiku-4-5", "Is this routine or complex? " + task)
    if triage.is_routine:
        return call_model("claude-haiku-4-5", task)   # cheap path

    # Tier 2: real work with Sonnet
    result = call_model("claude-sonnet-4-6", task)
    if result.confidence >= 0.9:
        return result

    # Tier 3: tough cases with Opus
    return call_model("claude-opus-4-7", task)

Real cost numbers from a tiered customer-support deployment:

vs flat-Sonnet at $0.20/run: tiered is 35% cheaper at the same overall quality. Worth the engineering for high-volume agents.

Sample size for A/B tests

Don't conclude from 20 runs. Statistical-significance ranges:

For low-volume agents (a paralegal doing 5 NDAs/day), the platform's A/B-test tool runs both configs in parallel, accumulating samples until significance threshold; then declares a winner.

A/B testing prompts (the CLAUDE.md A/B)

Same shape as model A/B, different variable. Run two CLAUDE.md versions on the same task fleet and compare:

• Variant A (current): existing CLAUDE.md
• Variant B (proposed): CLAUDE.md with the new constraint or example

Allocate 50/50 of the next 100 runs across A and B. Compare the four metrics. Ship the winner. Discard the loser.

Most prompt A/Bs are about reducing escalation rate, not improving success rate. "If the invoice is ambiguous, prefer to flag rather than guess" reduces 'wrong outputs' faster than it improves the success rate of clear-cut cases.

Real prompt A/B examples

Cohort comparison: same task, different agents

If multiple agents do the same workflow (e.g. you have 3 AI paralegals across 3 partners), compare them:

Why does Partner C's agent perform worse? Three possibilities:

1. Real-estate work is genuinely harder than litigation. Some workflows are more agent-shaped than others.
2. Partner C's CLAUDE.md is less specific. Compare the docs side by side; copy what's working in A and B.
3. Partner C is assigning more ambiguous tasks. The escalation rate hint at this.

Cohort comparison turns "is my agent good?" into "is my agent good compared to peers?" — a much sharper question.

Cross-fleet cohort comparison

For larger orgs with the same role across multiple departments (e.g. a Finance AP-clerk and an Operations AP-clerk), cohort comparison surfaces:

• Best practices that one team discovered and the other hasn't (copy from A to B)
• Workflow differences that justify different metrics (Finance pays more vendors; Ops fewer but larger)
• CLAUDE.md drift between teams (over time, two CLAUDE.md's diverge; cross-team review catches it)

The 30-day review template

Run this monthly per agent during the first 90 days; quarterly afterward. Takes 15-30 minutes:

When to retire an AI agent

Hard signals to retire:

• Success rate under 80% after 3 rounds of CLAUDE.md improvement
• Cost-per-task exceeds the human alternative when correctly accounting for human time
• Escalation rate above 15% — the work isn't agent-shaped; you're using the agent as a filter for human work, which has its place but should be priced as such
• Quality complaints from downstream consumers (regulators, customers, partners) that you can't reliably fix
• A better harness emerged. Devin → Claude Code, Codex → Cursor Cloud — keep the option open
• The work itself stopped existing. Companies pivot; some workflows go away. Retire the agent that handled them.
• Compliance posture changed. New regulation says this work needs human-only handling — retire the agent.

"Retire" doesn't mean "fire forever." It means: pause this configuration, write down what didn't work, try a different harness or different scope. Sometimes a retired agent comes back successfully on a narrower scope.

The retire-and-re-deploy pattern

1. Pause the current agent. Audit log captures everything; sandbox stops.
2. Document what didn't work. CLAUDE.md history; specific failure patterns; what was tried.
3. Decide the next-best option:
   • Different harness (Claude Code → Devin if more autonomy is needed; Devin → Claude Code if less)
   • Different model (Sonnet → Opus if reasoning was the bottleneck)
   • Narrower scope (instead of "all NDAs," just "mutual NDAs over $X")
   • Hand back to a human (some work is genuinely human-only)
4. Onboard the replacement with the same playbook (see Onboarding an AI Employee) — but skip steps the predecessor's history makes obvious.
5. Run side-by-side for a week if possible (replacement on shadow data; predecessor on production). Verify replacement is actually better, not just different.
6. Switch over; archive predecessor's CLAUDE.md and audit log per compliance retention.

The quarterly workforce review

Once a quarter, zoom out from per-agent to fleet-level. This is the meeting you have with your CFO / VP Eng / leadership team.

• Total fleet cost vs total fleet hours saved → fleet-level ROI
• Headcount equivalents — at current hours saved, how many human FTEs is the fleet equivalent to?
• Distribution by department — which departments are getting the most leverage? Where to invest next?
• Failure clusters across agents — same downstream system breaking 5 agents = a vendor issue, not an agent issue
• Compliance / audit posture — any new requirements that need new guardrails?
• Investment recommendations — which roles to add? Which to expand? Which to retire?

The output is a one-page report. Forward it to the CFO and the leadership team. The numbers are dramatic; the leverage is the story.

The one-page quarterly report template

# AI Workforce Quarterly Review — Q2 2026

## Headline
- 47 AI workers across 7 departments
- Total compute + model cost: $4,820 / quarter
- Hours saved (vs human alternative): 2,340 hours / quarter
- Headcount equivalent: ~6 FTE-equivalents
- Cost ratio: 1 AI dollar saves $14 of human time

## What changed
- Added: 3 paralegals (Legal); 2 customer-support tier-1 (Support)
- Retired: 1 sales-research agent (escalation rate 22%; not agent-shaped)
- Swapped: AP fleet moved Sonnet → Sonnet+reasoning; +12% success, +28% cost

## By department
| Dept | Agents | Cost | Hours saved | ROI |
| ---- | ------ | ---- | ----------- | --- |
| Finance | 8 | $620 | 480 | 38× |
| Legal | 12 | $1,840 | 720 | 17× |
| Support | 9 | $480 | 540 | 49× |
| Marketing | 6 | $720 | 320 | 19× |
| Eng | 7 | $560 | 180 | 14× |
| HR | 3 | $200 | 80 | 17× |
| Ops | 2 | $400 | 20 | 2.5× ⚠️ low


## Investment recommendations next quarter
- Expand Finance fleet (highest ROI; bookkeeper backlog growing)
- Investigate Ops fleet ROI (2.5× is low; either work is wrong-shaped or CLAUDE.md needs work)
- Retire the legacy "social-media-monitor" agent (volume dropped 80% YoY)

## Compliance posture
- All audit logs at 7-year retention (SOC2 + financial)
- Per-agent encryption layers documented
- Quarterly access-review completed for all 47 agents

Ship one of these per quarter. Forward to the CFO and the leadership team. Don't make them ask for it.

Industry benchmarks (anonymized)

Steady-state ranges from operators we work with at month 4+ of agent operation:

If your agent is well below the 25th percentile of its role's range, the playbook for getting to the 50th is usually CLAUDE.md tightening. If it's above the 75th, you're operating well — focus on scaling rather than tuning.

Common review mistakes

1. Reviewing too often

Operators new to AI workforce do weekly reviews because they're anxious. Weekly is too often after the first 90 days; you're picking up noise instead of signal. Drop to monthly after first 90 days, quarterly after first year.

2. Reviewing one agent in isolation

"Is this agent good?" is a worse question than "is this agent good compared to peers in the same role?" Always do cohort comparison if you have multiple agents in the same role.

3. Optimizing for one metric

Improving success rate by tightening escalation rules increases escalation rate, which decreases success rate. Tracking only one metric leads to gaming. The four metrics together are the framework.

4. Not setting a "what done looks like" before A/B testing

"Test if Opus is better than Sonnet" with no specific success criterion = an A/B that never concludes. Pick the criterion (success rate up 3%; cost-per-task down 20%) before you start; declare based on the data hitting the criterion.

5. Treating model upgrade as automatically good

"GPT-5.5 is out, let's upgrade." For some tasks, yes. For most, run an A/B for 2 weeks and let the data decide. Free upgrades sometimes regress on specific task classes.

6. Skipping the substance review

Process metrics are easy. Substance review is harder. The agents that look good on process but ship subtly-wrong work are the ones you don't want. Make substance review a quarterly habit even if metrics look fine.

7. Forgetting to document the decision

3 quarters from now, no one remembers why you swapped from Sonnet to Opus. Write it down — one paragraph in the CLAUDE.md history. Audit-log-as-decision-log.

Communicating ROI to the CFO

The conversation goes one of three ways depending on framing:

The third framing wins. The math is identical; the story is dramatically different. Use it.

Concrete numbers that work in CFO conversations:

• Cost-per-task vs human alternative ("paralegal at $15 per NDA vs AI at $0.45 per NDA")
• Hours-saved-per-month and what the freed-up human does instead ("paralegals are now spending 70% of their time on billable client work, up from 30%")
• Throughput increase ("we used to file 50 motions a quarter; we now file 200")
• Quality improvements ("citation error rate dropped from 1.2% human-only to 0.3% with AI verification")
• Capacity unlock ("we can now take on 3× the case load without hiring")

Numbers like these turn the AI workforce from a cost line into a capability narrative. CFOs care about the latter.

Frequently asked questions

How often should I run a performance review?

Per-agent: monthly during the first 90 days, then quarterly. Per-fleet: quarterly. Per-model-or-prompt-A/B: ad-hoc when you're considering a change.

Do I review the model or the agent?

The agent. The model is a component, like the type of laptop a human uses. The agent (job description + tools + model + workflow) is the unit of work.

What if my agent's metrics are good but the team complains?

Spot-check 10 runs and ask the complainers to sit with you. Often the metric is "completed without escalation" but the team's complaint is "the output isn't quite right and I'm fixing it silently." If complaints exist, the metrics are missing a dimension — usually quality. Add a sampled quality review (rate every 20th run).

Can I compare AI agents to human employees in performance reviews?

Yes, on output and cost — not on growth potential. Humans get raises and stretch goals because they grow. AI agents stay at their current capability (the model improves on a separate cycle). Don't blend the review formats.

What's the right sample size for an A/B test?

Big effect (10%+): ~100 per arm. Medium (3-5%): ~500 per arm. Small (1-2%): 2,000+ per arm. Tiny (under 1%): usually not worth chasing.

How do I review an agent doing work I don't fully understand?

Split into process and substance. Process metrics you can review. Substance review needs a senior in-domain human. Schedule monthly substance review with a 5-run sample.

When should I switch from one model to another?

When the data says so, not when the news says so. New model launches don't automatically warrant a switch. A/B for 2 weeks; switch if meaningfully better at acceptable cost.

How do I communicate the AI workforce ROI to my CFO?

Three numbers per fleet: total fleet cost, total hours saved, headcount-equivalent ratio. Frame as "humans now manage 5-15 AI workers" not "AI replaced humans."

What happens if the model degrades after a model update?

The platform supports model version pinning. Pin to a specific version for production agents; upgrade deliberately, not automatically. A/B for a week before switching the full fleet.

Should I review every agent the same way?

No. Mission-critical: monthly with substance spot-checks. High-impact internal: quarterly. Low-impact internal: semi-annual. Always-on monitoring: only on metric anomalies.

What's the right number of agents per supervisor?

5-15 at sustainable cadence. 5 if high-touch (paralegal, code reviewer). 15 if low-touch (AP clerks, monitoring). Past 20 per supervisor, supervision quality drops.

Can I see how my agents compare to industry benchmarks?

Yes, anonymized. The platform publishes quarterly benchmarks by role showing 25th/50th/75th percentile.

What's the difference between performance review and supervision?

Supervision is daily/weekly — what's happening now. Performance review is monthly/quarterly — how is the agent doing over time. Both are needed.

How do I retire an agent without losing the work?

Three options: (1) replace with a different harness or model — same role, different tools, run side-by-side for a week. (2) re-scope to a narrower role and retire the broader version. (3) hand back to a human if the AI version genuinely doesn't work.

What if my agents' metrics are flat — neither improving nor degrading?

Flat is healthy at steady state. After the first 3 months, you should see relatively flat metrics — that's the agent operating predictably. Don't over-tune.

What's next