A Successful Pilot Is Not Yet Successful AI Adoption

A pilot proves a principle. It shows that a given method, on given data, works for a given task. That is an important precondition for whatever comes next — and it is where the real work begins.

Between a first proof of concept and routine day-to-day operation there is a good deal of ground to cover. In most pilots, nobody plans that stretch in any detail. Six stages stand between the pilot and a finished project, and each one costs time and attention. Any of them, on its own, can bring the project to a halt. So it pays to be clear about what still lies ahead once the pilot has succeeded.

Seen that way, it makes sense that pilots scoped too narrowly, or thought through too briefly, leave a lot of questions open — and may put the wider project at risk. Which is an argument for thinking about all this while the pilot is still being designed.

1. From Test Data to Real Data

A pilot often runs on data created or gathered for the demonstration. Production shows what actually comes through the door: records with gaps, unexpected legacy holdings spanning three generations of systems, and the like. These are ambiguities that were absent from the test set, and could not have been spotted there.

This stage is pure data work, and it not infrequently takes longer than the pilot did. Building a pilot on idealised data because it makes the proof easier does not spare you the stage; it defers it. And later work to extend the solution may well cost more — particularly where the problem does not surface straight away.

If the pilot cuts this short: the model meets a data reality it has never seen. Quality degrades slowly enough to go unnoticed, until someone further down the line passes on an answer that is wrong.

2. From Single Case to Process Variants

The pilot implements one process. In the organisation that process may well exist in variants: by site, by product line, by long-standing exception introduced years ago for good reason.

Every variant is a decision. Do we support it, standardise the process, or deliberately leave it out? These are business decisions, not technical ones, and they need the people who own the process in question.

If the pilot cuts this short: it was optimised against an idealised requirement. Rolling out the real thing runs into variants, and it is quite possible that nobody has settled who decides how to handle them. It is not unusual for the roadmap to stall while the necessary — and unforeseen — discussions run their course.

3. From Demo to Production Readiness

Pilots often skip what production has to satisfy: authentication, audit-proof logging, access boundaries, integration with existing data flows, consumers for the output. On top of that comes everything a system needs in order to be observable — monitoring, alerting, and a fallback for when a new version turns out worse than the old one.

This work is well understood and can be planned. It simply tends not to be commissioned, because it never came up in the pilot.

If the pilot cuts this short: the follow-on programme needs a budget that the pilot’s success does not automatically justify, and that often was not planned for in the current year.

4. From Individual Sponsor to Operational Ownership

A pilot often has a sponsor somewhere in the organisation. Production needs a structure that can carry ownership of the system: for availability, cost, quality drift, and behaviour when things go wrong. It has to be wired into existing processes, and a new system needs a budget line of its own. It would not be the first time a good idea, technically sound and ready to go, foundered on organisational structures that were never put in place.

If the pilot cuts this short: more often than not, the conversation about building that structure only starts once the pilot ends. Who owns this from now on? Whose remit does the new system get added to? What budget does it need, and which pot does it come out of?

5. AI Is Different: What Is It Allowed to Decide?

This is where AI parts company with conventional software. A traditional system carries out what somebody has already decided. An AI system produces a judgement — a recommendation, a classification, a draft — and that puts a question on the table nobody had to ask before: who is accountable for that judgement?

Several commitments hang together here. Who is allowed to overrule the output, and who is actually obliged to review it? Above what level of consequence must a person decide? Who can stop the system when it is plainly wrong, and whose sign-off do they need? And who gets told that a judgement was made by a machine at all — including the people on the receiving end?

These questions touch liability, duties of oversight, and in regulated settings documentation requirements too. Left open, every department draws the line somewhere else: some review every output and lose the benefit, others wave everything through and carry an exposure nobody has put on the books.

If the pilot cuts this short: during the pilot every result was looked at and judged — often informally, because there were only a handful of cases. Across a full production workload that no longer holds, and is not practical. This difference matters, and it is frequently misjudged.

6. From Trial Run to Steady State

Being available is not the same as being used, and usage can still shift once it is. People change how they work when they trust a system and start relying on it being there. Processes often change, methods get replaced, ownership moves on. Meanwhile the technology keeps developing, and once systems are running on new models the question arises: who checks that the system still does what it should? Who decides on the next model version, and on what basis? This kind of quality assurance often has no deadline forcing it to happen — which is why it is the thing most likely to be dropped.

If the pilot cuts this short: pilots are often run with staff who have a stake in the pilot succeeding and who like the new method anyway. Where nobody has defined how performance is to be measured, production tends to be missing both: anyone whose job it is to check quality, and a yardstick that would show results getting worse in the first place.

How the Pilot Is Designed Shapes Whether It Succeeds

Six stages, all of them still to be worked through once the pilot ends. This is not an argument against testing feasibility by running a pilot, but it should be clear that a pilot is bound to leave a good many questions open. There is a strong case for designing pilots with the build-out and implementation phase already in view.

The effort is not the same for every pilot, and the distance to production is not always equally long. A pilot that picks the cleanest slice of data, the simplest site and the most willing group of users proves its principle sooner — and leaves all six stages ahead of it at full length. One that deliberately takes on middling difficulty — real data, a site with the usual deviations, users who did not ask for any of this — takes longer and costs more, but it shortens everything that follows.

In practice that means setting the thresholds for production before the work starts: response time, a floor on accuracy, cost per call, behaviour on failure. It means naming who will be responsible at the outset and preparing the organisational structure early, rather than turning to the question of how production will be supported only when a finished project is handed over. If an early discussion turns up a list of things nobody has the time or the money for, then the pilot is not testing feasibility in this organisation. It is only testing whether a demonstrator can be built.

How well a pilot ends up carrying the project that follows comes down to how it was scoped, and to the questions that can be settled early — at a point where neither costs very much.