In short
- Pilots are usually built to impress. Production systems are built to be run.
- The model is rarely the blocker. Data access, ownership and an agreed measure of "good" usually are.
- AI in daily use has a named owner, a test set it must pass, and a way for people to correct it.
- Start with one frequent, costly task, and plan the handover to operations from the first week.
A promising AI demo takes a week. A few months later, the same idea is often still called a pilot. It has a slide in the steering committee pack, a small group of enthusiastic users and no date for going live.
This is so common that it’s easy to blame the technology. In our experience it’s rarely the model. The pilots that stall and the ones that reach production usually use similar models. What differs is everything around the model: the data it can reach, who owns it, how quality is judged and how the work is run once the pilot team moves on.
The demo answers a different question
A demo answers one question: can a model do this task? For most business tasks today, the answer is yes, at least some of the time.
Production asks harder questions:
- Can we rely on it every day, for every user, at a cost we accept?
- How do we know it’s still right next month?
- Who fixes it when it’s wrong, and how quickly?
- What happens to the people and systems around it?
A pilot that was only designed to answer the first question has no answers to the others. That’s where it stalls.
Five reasons pilots stall
1. Nobody owns it after the pilot. The pilot was run by an innovation team or a small squad. When they move on, there’s no business owner accountable for results and no engineering team responsible for keeping it running.
2. The data worked in a sandbox, not in the business. The pilot used a copy of the data, cleaned by hand. Production needs live data, the right permissions for each user and data that stays fresh. Those take real engineering.
3. There’s no agreed measure of “good”. Without a test set of real cases and an agreed way to score them, every discussion about quality comes down to anecdotes. One bad answer in a meeting can stop a project that is right most of the time.
4. Risk and security arrive at the end. Legal, security and risk teams see the system for the first time when it’s ready to launch. Their questions are reasonable, but answering them late means rework.
5. The work around the model didn’t change. People have to copy an answer out of one tool and paste it into another. The AI saves time in theory and adds steps in practice, so people quietly stop using it.
What the ones in production did differently
The teams that get AI into daily use tend to share a handful of habits.
- They picked a narrow, frequent, costly task. Not “transform customer service”, but “draft the first reply to billing questions”. Frequent tasks give fast feedback, and costly ones make the value easy to see.
- They named two owners. A business owner who is accountable for results, and an engineering team that runs the system like any other production service.
- They wrote down what good looks like before building. They collected real examples, agreed how to score answers and set the bar the system had to clear before each release.
- They designed people into the workflow. Reviewers approve, edit or reject the AI’s work inside the tools they already use, and every correction is captured.
- They brought in risk and security in week one. Data handling, access and audit questions were part of the design, not a launch blocker.
- They tracked cost per task. Model and infrastructure costs were measured against the value of the task, so scaling up was a business decision, not a leap of faith.
Pilot thinking and production thinking
| Question | Pilot answer | Production answer |
|---|---|---|
| Who owns it? | The team that built the demo | A named business owner, and an engineering team that runs it |
| Where does the data come from? | A copy, cleaned by hand | Live, governed access with the same permissions as the user |
| How is quality judged? | It looked right in the demo | Scored against an agreed test set before every release |
| What happens when it’s wrong? | Someone notices | Users flag it, and flagged cases join the test set |
| What does it cost? | Not tracked | Cost per task, tracked against the value of the task |
| When is risk involved? | At launch | From the first week |
Agents raise the stakes
Agentic AI goes a step further: instead of drafting an answer, an agent takes actions in other systems. It updates a record, raises a ticket or sends a message. That makes the production questions more important, not less.
Before an agent acts on its own, decide:
- which actions it may take, and with whose permissions;
- which actions need a person to approve them first;
- how every action is logged, so it can be audited and explained; and
- how an action is reversed if it was wrong.
Start agents on actions that are easy to check and easy to undo. Widen their scope as the evidence builds.
Where to start
- Pick one workflow. Choose a task that happens often, takes real time and has a clear owner.
- Agree what good looks like. Collect real examples and decide how answers will be scored.
- Connect to real data early. Use the permissions and data sources the live system will use.
- Ship to a small group with review built in. Let people approve, edit or reject the output where they already work.
- Measure and widen. Expand to more users when quality and cost hold up against the test set.
None of this is glamorous. But it’s the difference between a demo people remember and a system people use every day.
How ManyaIT helps
Our applied AI and agentic AI teams build this way from the first sprint. They work inside your environment, matched to your stack, with evaluation, monitoring and handover planned in from the start. Tell us the workflow you want to move into production, and we’ll map the route from pilot to a governed service.
