Two true lines
One dispute overlay, first quarter in production · indexed to week 1 = 100
Week in production
Every AI program we audit dies one of two deaths at review time. The first team walks in with an accuracy dashboard — the model is right 94% of the time, look at the confusion matrix — and the CFO asks what changed in the P&L, and nobody has an answer denominated in money. The second team has a genuinely working overlay, never metered the spend, and walks in a week after finance found a token invoice that tripled in a quarter. Different funerals, same cause: the wrong numbers on the page.
The fix is not more dashboards. It is a small, fixed scorecard — we install nine numbers, in three layers — reviewed weekly with the same discipline as any other operating metric. This memo is that scorecard, including the thresholds that should trigger a scale-up, a fix, or a kill.
Layer one: queue KPIs — the numbers that justify existence
An AI overlay exists to make a queue clear faster, at lower cost, with controls intact. So the first layer is the same set any operations leader already respects, measured against the pre-overlay baseline:
- Cycle time per case — intake to resolution, median and P90. The headline number.
- Touches per case — how many times a human handles the item before it closes. This is where drafting-quality problems show up long before anyone admits them.
- Backlog age — oldest open case. Overlays that cherry-pick easy cases look great on cycle time and rot the tail.
- SLA hit rate — because a regulated queue that got faster on average but started missing scheme deadlines did not improve.
Nothing exotic. The discipline is in what's not here: accuracy, F1, benchmark scores. Those are diagnostics for the engine room — when a queue number degrades, you go look at them. They never go on the leadership page.
Accuracy is a diagnostic, not a KPI. The queue doesn't care how right the model is — it cares how fast a case leaves, at what cost, with which controls intact.
Layer two: cost meters — cost per case, not cost per token
Per-token prices are the most quoted and least useful numbers in this market. A token price tells you nothing until you know how many tokens a resolved case consumes — and that number moves constantly, in both directions, for reasons that have nothing to do with the vendor's price list.
- Cost per resolved case — the master metric: model spend plus the human review minutes the case still needs, divided by cases closed. Price it against the fully loaded labor cost it displaces. Our working rule: if AI cost per case creeps past 10–20% of the displaced labor cost, review overhead is eating the margin and something is wrong upstream.
- Tokens per resolution — the trend, not the level. A silent 30% climb over six weeks is a workflow defect: retry loops, context stuffing, an agent circling. This is the meter that catches budget fires while they are still cheap.
- Cache and reuse rate — the share of input tokens served from cache or templates. On document-heavy queues this single number often moves the bill more than any model choice.
The chart above is the shape of health: absolute spend growing with volume while unit cost falls with maturity. Budget the first line, manage the second. A token budget set as a flat monthly cap punishes success — the correct cap is per-case, per-queue.
The three burn patterns that eat budgets
Almost every runaway token bill we've been called to autopsy was one of three patterns, each visible weeks earlier on one meter:
- The loop. An agent retries or self-corrects in circles; tokens per resolution climbs while queue metrics stay flat. Cap steps per case, alert on the trend, and read the traces of your ten most expensive cases every week — the top ten are never random.
- Frontier by default. Every request routed to the most capable model "to be safe." Route to the cheapest tier that passes QA on that case type and escalate on failure; on most operational queues the workhorse tier handles 80%+ of volume. This is routinely a 5–10× unit-cost difference for zero queue impact.
- Context stuffing. Whole case files pasted into every prompt because retrieval was the deferred workstream. Shows up as a high, flat tokens-per-case with excellent accuracy — the most expensive way to be right. The cache-rate meter exposes it.
Layer three: guardrail KPIs — the numbers the regulator asks about
- Override rate — how often a reviewer rejects or materially edits the AI's output. Falling steadily: trust is being earned. Flat and high after week four: the model is wrong, the gates are wrong, or the task was never automatable — find out which before scaling anything.
- Escalation integrity — the share of mandatory-escalation markers (AML flags, RG signals, scheme deadlines) that reached a human on time. The target is 100%, and the honest version is measured by sampled audit, not by the system grading itself.
- QA pass rate — a weekly random sample of closed cases re-reviewed by a senior human. This is the number that keeps the override rate honest: if overrides fall while QA failures rise, reviewers have stopped reading.
The scorecard on one page
| Healthy | Act when | |
|---|---|---|
| Cycle time per casequeue · median + P90 vs baseline | Falling 30–60% by week 8 | Flat after week 4 — fix the workflow, not the model |
| Touches per casequeue · human handles per item | Trending toward one | Rising — drafting quality is slipping |
| SLA hit ratequeue · deadlines met | At or above baseline from week 1 | Any regression — pause the rollout, no debate |
| Cost per resolved casecost · tokens + review labor | Falling as volume grows | Above 10–20% of displaced labor cost for 2 months |
| Tokens per resolutioncost · trend line | Flat or falling | +30% over 6 weeks — hunt the loop or the stuffing |
| Cache / reuse ratecost · share of cached input | Rising with maturity | Low and flat on a document queue — retrieval is missing |
| Override rateguardrail · reviewer rejections | Falling steadily from week 2 | Flat and high at week 4 — stop scaling, diagnose |
| Escalation integrityguardrail · sampled audit | 100%no other number is acceptable | Anything else — halt automation on that case class |
| QA pass rateguardrail · weekly senior sample | Stable while overrides fall | Falling while overrides fall — reviewers stopped reading |
Decide the thresholds before week one
The scorecard only works if the responses are agreed before emotions and sunk costs arrive:
- Scale a queue when cost per case and override rate have both fallen for four consecutive weeks with SLA at baseline. Scaling earlier buys volume on an unproven unit economic.
- Fix when queue numbers stall but guardrails hold — that's a workflow problem, and it is almost never solved by upgrading the model. Re-read the traces, move the gates, fix retrieval.
- Kill on the criteria you wrote down at kickoff: escalation integrity below 100% twice, unit cost above the labor it displaces at month two, or an override rate that never learned. A killed workflow returns budget; an unmeasured one salts the whole program.
One page, nine numbers, every Tuesday, same table leadership already reads for every other operation. That's the entire trick: AI stops being a science project the week it starts being reported like a queue.
Frequently asked questions
What KPIs should you track for AI in production?
Three layers: queue metrics against the pre-AI baseline (cycle time per case, touches per case, backlog age, SLA hit rate), cost meters (cost per resolved case, tokens per resolution, cache rate), and guardrails (override rate, escalation integrity, QA pass rate). Model accuracy is a diagnostic for engineers, not a leadership KPI.
How do you keep LLM token costs under control?
Meter cost per resolved case, not cost per token. Set per-case, per-queue budgets instead of flat monthly caps, route work to the cheapest model tier that passes QA and escalate on failure, invest in caching and retrieval instead of pasting whole case files into prompts, and alert on the tokens-per-resolution trend — a 30% climb over six weeks is a defect, not growth.
What is a good cost per case for AI?
Price it against the fully loaded labor cost the automation displaces. As a working rule, AI cost per resolved case — model spend plus remaining human review time — should sit below 10–20% of that displaced cost. Above that line, review overhead is consuming the margin and the workflow needs fixing before it needs scaling.
When should you kill an AI workflow?
On thresholds agreed before launch: mandatory escalations missing a human more than once, unit cost still above the displaced labor cost after two months, or an override rate that stays high past the first month. Killing on pre-agreed numbers returns budget and credibility; letting an unmeasured workflow drift costs both.