One engagement, week by week
Representment backlog during a ten-week rollout
Engagement week
The demo was flawless. The model summarized a real dispute case in eleven seconds, the CFO said "this changes everything," and the innovation team got budget for a pilot the same week.
Fourteen months later we were in the same boardroom doing an audit. The pilot had been "in evaluation" for over a year. Nobody could say who owned it. The vendor contract had auto-renewed twice. And the dispute queue it was supposed to transform was being worked exactly the way it was before the demo — by hand, in a spreadsheet, with the pilot running quietly in a sandbox next to it.
This isn't a story about one company. It's the default outcome. When we start an engagement, the first thing we inventory is what's already been tried — and at a typical mid-size regulated operator we find between four and nine AI initiatives in various states of undeath. Demos that impressed everyone. Pilots that proved the concept. Production launches that never happened.
We call it the pilot graveyard, and it has almost nothing to do with the models.
Where AI initiatives stall
Three ways a pilot dies
Autopsies get repetitive. Across the initiatives we've audited, nearly every death certificate lists one of three causes.
1. It was measured on the model, not the queue
The pilot's success metric was accuracy: the model classified 94% of tickets correctly, extracted the right fields from 96% of invoices. Everyone agreed the technology worked.
But nobody rebuilt the workflow around it. The queue still routed every item to a human first. The human still did the work the old way, because the pilot's output landed in a separate tab they had to remember to check. Accuracy was proven; throughput never changed. After six months, the finance team asked the obvious question — what did we get for this? — and there was no answer denominated in anything they cared about.
A model that is right 94% of the time and changes nothing about how work moves is worth exactly zero. The unit of progress is not accuracy. It's a queue that clears faster with the same controls.
2. It had a sponsor, but no owner
Every dead pilot had an executive sponsor — someone senior who said the word "strategic" in the kickoff. Almost none had an owner: a named person inside the operating team whose actual job, on their actual scorecard, was to get the thing into the daily workflow.
The innovation team can't own it; they don't run the queue. The vendor can't own it; they can't change your SOPs. So the pilot floats. It works, technically, and it belongs to nobody, operationally. When the sponsor changes roles — and in fourteen months, they will — the pilot doesn't get killed. It gets orphaned, which is worse, because orphans still consume budget and still show up on the risk register.
3. Compliance met it last
In a regulated business this one is fatal and completely predictable. The pilot was built in a sandbox precisely so it could move fast — no PII, no production access, no sign-offs. That was the point. But it also meant the first time compliance saw the system was after the business had decided it wanted it.
So compliance did its job. It asked how the model's decisions would be evidenced to the regulator, what happens to the audit trail when a human accepts an AI-drafted representment, who reviews the edge cases. The pilot had no answers, because answering those questions was deferred to "the production phase." The production phase is where those questions live. That's why it never started.
A pilot that cannot inherit your audit trail was never going to production. It was theatre with a GPU budget.
What the survivors have in common
Some initiatives do cross the gap. In our portfolio, the ones that made it share a shape — and it's the opposite shape of a pilot.
| The overlay | The pilot | The rebuild | |
|---|---|---|---|
| Runs against | The live queuefrom its first build week | Sandbox data | A staging stack, someday |
| Measured on | Cycle time, cost per case, SLA | Model accuracy | Milestones shipped |
| Owned by | The team that runs the queue | Innovation team | A program office |
| Compliance | In the room at design | Consulted at the end | Re-certifies everything |
| Human sign-off | Where the regulator expects it | Bypassed in the sandbox | Redesigned from scratch |
| Audit trail | Richerevery draft logged with its reviewer | Deferred to production | Rebuilt from zero |
| Leadership sees | Queue metrics, weekly | A demo, once | A Gantt chart, quarterly |
| Ends with | A changed SOP and a handover | A readout deck | A two-year freezeif it ends |
We use the word overlay deliberately. The survivors don't replace the workflow and don't run beside it — they sit on top of the existing queue, inside the existing controls, and take over one step at a time. The human sign-off stays where the regulator expects it. The audit trail gets richer, not thinner, because every AI-drafted action is logged with its inputs and its reviewer.
That's also why the survivors are boring to demo. There's no eleven-second magic moment. There's a Tuesday where the representment backlog is 40% shorter than the Tuesday before, and a compliance officer who can explain exactly why to anyone who asks.
The cadence that replaces the pilot
The practical alternative to piloting isn't recklessness — it's operating cadence. Ours runs ten weeks in four phases — discover, diagnose, overlay, compound — and the sequence matters less than the rules it enforces:
- Discovery happens inside the queue, not in a data extract. The first weeks are interviews with the people actually working the cases — where the real volumes, edge conditions, and workarounds live. Sandboxes lie, and so do process docs.
- The owner is named before the kickoff. One person, inside operations, with the metric on their scorecard. If the business can't name that person, the engagement isn't ready — that's a finding, not an inconvenience.
- Compliance designs the gates before the overlay exists. Which actions the system may draft, which it may execute, which evidence gets attached to each — decided during diagnosis, with the people who will defend it to the regulator.
- The overlay never sees a sandbox. From its first build week it drafts read-only against live cases, inside the existing controls, and earns each next permission with numbers.
- Every week ends with a number. Cycle time, touches per case, backlog age — reviewed with leadership weekly. Not model metrics. Queue metrics.
- Hand-off is the deliverable. The engagement ends when the client's team runs the overlay without us — SOPs updated, on-call named, evidence pack templated. A readout deck is not an ending.
None of this is exotic. It's the same discipline any operations leader applies to a process change. The industry suspended that discipline for AI because the demos were so good — and the graveyard is the invoice for that suspension.
When killing a pilot is the right call
Not every stalled initiative deserves rescue. We recommend a deliberate kill when we see any of these:
- The use case has no queue. If the work it touches isn't a repeated flow with volume and an SLA, an overlay has nothing to hold onto. One-off analyses make fine tools and terrible transformations.
- The unit economics fail at list price. If the case only works with pilot-tier pricing, it dies at renewal anyway. Better now.
- The control question has no answer. Some actions genuinely can't be evidenced to the regulator's standard yet. Write that down, park it, revisit in two quarters — don't let it rot in "evaluation."
A killed pilot returns budget and attention. An undead one consumes both and salts the ground — every future initiative inherits the skepticism it earned.
The uncomfortable summary: the pilot phase, as most enterprises run it, is a mechanism for deferring the operational questions that decide whether AI ships. The model was never the risk. The workflow, the ownership, and the audit trail were — and they're only testable in production, under controls, on the live queue.
Stop piloting. Start operating.
Mikhail Landau — CEO, ORG