The Metrics That Predict Agent Failure Before Customers Do

An agent rarely fails loudly. It degrades quietly, one slightly worse output at a time, until a customer notices before you do. The good news is that degradation is measurable well before it becomes visible externally. A small set of operational metrics — quality-gate pass rate, escalation rate, drift indicators, and cost-per-successful-task — function as an early-warning system. The operating cadence is the rhythm that reviews an agent portfolio; these are the numbers that rhythm actually watches. This post is about the measurement, not the calendar.

The Four Numbers Worth Watching

Quality-gate pass rate is the tripwire. When you point an automated scanner at every output — the same principle as running a code quality scan on generated code — you get a continuous pass/fail signal instead of a periodic opinion. A codebase scanned this way produced 406 findings in 35.2 seconds; the value was not the count but the speed. A pass rate that slips from 95% to 88% over a fortnight is telling you something is wrong long before any single bad output reaches a customer.

Escalation and exception rate is the second number. Every well-run agent has a path to hand a task to a human when it is uncertain. The rate at which it does so is a health reading. A rising escalation rate means the agent is meeting inputs it can no longer handle confidently — a genuine early warning. A falling one is more dangerous: it can mean the agent has stopped recognising when it is out of its depth and is now failing silently.

Drift indicators track the gradual degradation described in the multi-venture operating context. Model updates shift behaviour, prompt sensitivity changes, and context errors accumulate. You catch this by holding a fixed set of reference tasks and re-running them on a schedule, comparing output against a known-good baseline. A widening gap on tasks the agent used to pass cleanly is drift, quantified.

Cost-per-successful-task is the metric that ties the rest together, because it is honest about failure in a way raw cost is not. Divide total spend — inference plus your own oversight time — by the number of outputs that actually cleared the gate. Inference prices fall every quarter, so if your cost-per-successful-task is rising while token prices drop, the agent is producing more rework, more retries, and more escalations per genuine success. The economics expose the degradation before the customer does.

Reading Them Together

No single number is sufficient; the signal is in the pattern across all four. A pass rate holding steady while cost-per-successful-task climbs points to expensive retries hidden behind an eventually-correct output. A falling escalation rate alongside a widening drift gap is the worst combination — an agent that is both getting worse and losing its ability to flag it. The measurement discipline is cheap to build: a gate on every output, a fixed reference set run on a schedule, and one spreadsheet that tracks the four figures over time. The cost of instrumenting this is trivial next to the cost of a customer discovering the failure for you.

This is practitioner truth, not theory. The same agent economics that make a portfolio of ventures viable — variable cost, near-zero marginal labour — only hold if the output stays trustworthy, and trust is something you measure or lose. Chapter 4 of The AI Agent Economy lays out the cost structure in full, including why cost-per-successful-task is the number that separates an agent business that compounds from one that quietly bleeds. Instrument the four metrics, watch them together, and you will see failure coming while it is still a trend line and not yet a complaint.