Why the pilot never becomes production

What a demo does not measure

At best, a pilot proves this: on a chosen set of data, on a chosen task, with a chosen person watching, the model can do it. Production removes all three choices. The data arrives unchosen, the task arrives with its edge cases, and nobody is watching.

So the question after a pilot is not whether the model is good enough. The question is whether the work still runs when the model is right ninety percent of the time. If it does, you can go to production. If it does not, the fix is not a better model – it is a redrawn process.

In most projects that distinction is the difference between two weeks and six months.

There is a way to see this early: build the pilot around the bad case rather than the good one. Hand the team the fifty worst documents from last month instead of the fifty cleanest. The demo is less impressive and the decision is better.

The data that arrives on a Tuesday

Pilots run on clean data because a person picked the pilot data. A real Tuesday looks different: a skewed scan, a contract missing two pages, the same supplier spelled three ways, a product code a spreadsheet decided was a date, and ten files in the wrong folder.

There is a cheap way to measure this. Pull two hundred random records from the last ninety days of real input, feed them in without any cleaning, and keep three numbers.

• How many records could not be processed at all. These are the visible failures, and the harmless ones.
• How many were processed but produced the wrong answer.
• How many of the wrong answers were silent: the system looked confident and was wrong.

The third number decides everything. A silent error also kills the reviewer's instinct to check, so its cost compounds. Above roughly two percent, it is too early to talk about a go-live date – the first job is making the system's confidence measurable.

Run that measurement with the client's own team in the room. Reading two hundred outputs together convinces more than a thirty-page evaluation, and it resets expectations: nobody is waiting for a hundred percent.

A system with no owner does not survive

A pilot always has an owner: the person who asked for it. Production frequently has none. The project closes, the team disperses, the system keeps running, and for six months nobody reads its output.

An owner is a person who spends an hour a week on the logs and has the authority to move a threshold. They do not need to be technical; they need to know the work. Without a name, the system falls out of use within about six months – not because it broke, but because nobody stood behind it.

The most useful question at a handover meeting is this: who reads this system's output next quarter? If the answer is a department name, there is no answer.

There is a cheap way to find the owner: put a fifteen-minute weekly slot in the calendar for the first month after handover. Whoever turns up is the owner. If no slot can be found, the system is not doing anyone's work, and learning that now beats learning it in six months.

Ownership is a budget question too. Whose line pays the model bill, the small fixes, the provider change that comes round each year? Those costs end with the project budget and begin in the operating one; systems that fall in the gap get switched off at the first invoice argument.

Pick the number before you build

Most pilots end on a feeling that it works well. A feeling is worth nothing in a budget meeting. Whoever defends the system needs one number, and that number has to have been measured before the pilot started.

The good numbers are narrow and dull: the average time between an invoice landing on a desk and reaching accounting; hours to first reply; the share of lines corrected by hand; how many quotes went out late last month. Model accuracy is not on that list, because nobody saves money on accuracy.

Keep the comparison honest, too. Measure the new system on its best week against the old process on its worst and you get a number nobody can defend six months later.

Measuring has a side benefit: pinning the number down shows which step is actually expensive, and sometimes the answer is not AI. At one client, three quarters of the delay sat in an email waiting for approval. No model would have helped; changing the rule did.

Failure modes and the way back

Production systems survive not because they do not fail, but because their failures are visible and reversible. Three parts cover it: a threshold, a queue, an undo.

When the model's confidence sits below the threshold, the output is not applied – it drops into a queue and a person looks at it. When an output that was applied turns out to be wrong, reversing it should be one button. Those two mechanisms buy more operational safety than five points of accuracy.

Then the provider side. What happens when the model provider has an outage? There are two reasonable answers: queue the work and process it later, or fall back to the old rule-based path. Systems without an answer spend outage day with five hundred records stuck in the middle.

The queue needs an owner as well, or it collects four hundred records in three days and nobody opens it. Put a daily ceiling on it: if the ceiling keeps being breached, the fix is the process, not a lower threshold.

The last twenty percent: integration

Once the model works, the remaining work looks small and almost never is. It has a fixed shape.

• Identity and permissions: which user sees which record, and who can approve which output.
• Mapping to the source system: field names, unit and currency conversion, fields allowed to be empty.
• Retries and duplicate protection: what happens when the same invoice is processed twice.
• Logging, alerting and on-call: whose phone rings when it breaks, and what they can do about it.

None of this has to sit in the pilot budget; none of it can be left out of the production one. In our experience, getting from a working prototype to a dependable service takes about as long again as the prototype did.

What finishes this work is not technical difficulty, it is a named owner. When integration sits between two teams, weeks disappear from the calendar. Handed to one person, most of it takes days.

A budget for latency and cost

In a pilot nobody watches the seconds or the cents. In production both are part of the product. Write down four numbers, then build.

If those four are not written down, the system is not ready for production, because there is no yardstick to say whether it is.

There is a quiet line item on the cost side: re-runs. A batch that breaks gets started again from scratch and the bill doubles. A pipeline that resumes where it stopped pays for itself in the first month.

The go-live checklist

This is the list we work through when we decide to turn a pilot into production. Ten lines, all of them verifiable in a day.

Model quality appears in none of those ten lines. The reason is simple: pilots do not stall because of the model.

And set a date. A pilot without a go-live date does not finish, it slows down – and six months on, the model has aged enough that you start over.