Why AI projects stall between the demo and production

If your AI project worked in the demo and has been "almost ready" ever since, the model is probably not the problem. A demo answers one question: can a model do this task at all? Production asks a different set of questions: can it do the task on inputs nobody curated, can anyone prove it, does it fit inside your systems and permissions, what happens when it is wrong, and who owns it next quarter.

Projects stall because the demo answered the first question so convincingly that nobody budgeted for the rest. The remaining work is mostly not AI work. It is evaluation, integration, error handling, review workflow, and ownership — ordinary engineering and ordinary management, applied to a component that is wrong some percentage of the time.

The short version, for the executive reading this: ask to see the evaluation set and the written acceptance criteria. If neither exists, the project does not have a finish line, and that — not the model — is why it is not finishing.

The rest of this article is the longer version: the ten failure points, how to spot each one early, and a gate to put in front of launch.

What a demo actually proves

A demo is a small number of inputs, chosen by someone who wanted the demo to go well, run once, with a person watching who would quietly re-run anything that looked odd. None of that is dishonest. It is what a demo is for.

But it means a demo is evidence of capability, not reliability. The distance between the two is where the schedule goes.

The ten places projects stall

1. Demo inputs were curated; real inputs are not

The demo used clean PDFs. Production receives scans of faxes, photos taken at an angle, forwarded email chains with the actual question three replies deep, and spreadsheets where someone merged cells for decoration. The demo used well-formed questions. Real users type fragments, paste in error messages, and ask two things at once.

Early sign: nobody on the team can show you a random sample of real inputs. Not the best examples — a random sample. If the team has been developing against the same 20 files for two months, the system is fitted to those 20 files.

Fix: pull a few hundred real inputs at random, before building anything further. Read them. The reading alone usually reshapes the project scope.

2. There is no evaluation set, so "better" cannot be measured

Without a fixed set of inputs with known-correct outputs, every change is judged by trying a handful of examples and forming an impression. One engineer improves the prompt for invoices and silently breaks it for credit notes. Nobody finds out for three weeks. The project oscillates instead of converging.

Early sign: status updates say "accuracy is looking much better" with no number attached, or with a number nobody can reproduce.

Fix: build the evaluation set first. A few hundred labeled examples drawn from the random sample above, stratified across the input types you care about, scored by a script anyone can run. This is the single highest-return artifact in the project; we have written separately about what to measure once the system is live.

3. There are no acceptance criteria, so "done" cannot be declared

Even with a score, someone has to say what score is good enough — and that is a business decision, not an engineering one. It depends on what an error costs, what the current human error rate is, and what review will catch.

Early sign: ask three stakeholders what accuracy is required and get three answers, or get "as high as possible."

Fix: write it down. For example: "at least 97% field-level accuracy on the evaluation set; no more than 1 in 500 documents with a wrong monetary amount that passes validation; 95th-percentile response under four seconds." Numbers like these will be argued over. That argument is the point — it is far cheaper to have it in week two than in month six.

4. Latency and cost were measured with one user

One request at a time, a large model is pleasantly fast. At fifty concurrent requests, queueing begins, time-to-first-token climbs, and either the user experience or the bill goes somewhere nobody planned. Long prompts assembled from retrieved documents make both worse.

Early sign: no load test exists. Cost projections are "tokens per request times requests per month" using the demo's short prompts rather than production-sized ones.

Fix: load-test at expected peak concurrency with realistic prompt lengths. Report median and 95th/99th percentile, not the average. Build the cost model from measured token counts on real inputs. If the answer is uncomfortable, this is the moment to consider a smaller model, caching, routing easy cases to cheaper paths, or whether an LLM is the right tool for every step.

5. Integration, authentication, and data access were assumed

The demo ran on a CSV export. Production needs a live connection to the system of record — which has an owner, a change-control process, rate limits, and a security team. It also needs an answer to the permission question: when the assistant retrieves documents to answer a question, does it retrieve only what this user is allowed to see? Retrieval that ignores the source system's access controls is a data leak with a chat interface.

Early sign: the team that owns the source system has not been in a meeting yet. Nobody can say which service account the AI component will run as.

Fix: put integration and identity on the plan as first-class work items with their own owners and dates. Enforce permissions at retrieval time, using the user's identity, not the application's.

6. There is no plan for when the model is wrong

It will be wrong. It will also time out, return malformed output, refuse a legitimate request, and occasionally be unavailable. A demo has no failure path because a person was the failure path.

Early sign: the architecture diagram has one arrow out of the model, and it points to success.

Fix: design the unhappy paths deliberately: schema validation on every output, bounded retries, business-rule checks (do the line items sum to the total?), a fallback when confidence is low, and a clear "I cannot do this one" route to a human. A system that is right 94% of the time and knows which 6% to hand off is usually more valuable than one that is right 97% of the time and gives no signal.

7. The human review workflow does not exist

"A person will review the output" is a sentence, not a workflow. Which person? In what tool? How many items per day, at how many minutes each? What do they see — the output alone, or the output next to the source with the relevant region highlighted? What happens to their corrections?

Early sign: the reviewers have not been consulted, and the review step has no screen design and no staffing estimate.

Fix: design review as part of the product. Route by confidence so reviewers see the items that need them. Capture every correction — it is free labeled data for the evaluation set.

8. Nobody owns it after launch

The demo was built by a motivated individual or an outside firm. After launch, someone has to watch quality, handle incidents, update prompts when inputs shift, and decide when to change models. If that someone is not named, the system decays quietly.

Early sign: the project plan ends at "go-live."

Fix: name the owning team before launch, and hand over the things that make ownership possible: the evaluation set, the runbook, the dashboards. If you are buying, building, or hiring, make post-launch ownership an explicit line in that decision.

9. Security and compliance arrive late

Security, legal, and compliance teams have reasonable questions: where does the data go, who is the sub-processor, how long is it retained, is it used for training, what is logged, can we reproduce a specific output for an auditor, how is prompt injection handled? If they first hear about the project a month before launch, the honest answer to "when can we ship?" becomes "after their review queue."

Early sign: the first security review is scheduled after the build is complete.

Fix: a one-page data-flow diagram in the first two weeks, reviewed by security and legal. Their constraints — for instance, that certain data cannot leave your environment — may change the architecture, and it is much better to learn that before building.

10. Prompts and models drift

Hosted models are updated and retired on the vendor's schedule. A prompt tuned against one model version can behave differently on the next. Meanwhile prompts get edited in a console, with no history and no test run.

Early sign: the model identifier in the code is an alias like "latest." Prompts are not in version control.

Fix: pin model versions. Keep prompts in the repository. Run the evaluation set on every prompt or model change, the same way you run unit tests on every code change. Track deprecation dates as you would any dependency's end-of-life.

An early-warning checklist

You can run this in a thirty-minute conversation. Each "no" is a future delay.

• Can the team show a random sample of real production inputs?
• Is there an evaluation set, and can someone run it while you watch?
• Is the current score a number, and is there a written target?
• Has the system been load-tested at expected peak concurrency?
• Is the cost estimate based on measured token counts from real inputs?
• Has the owner of each source system agreed to the integration?
• Does retrieval respect per-user permissions?
• Is there a defined behavior for low confidence, invalid output, and timeouts?
• Have the human reviewers seen and used the review interface?
• Have security and legal reviewed the data flow?
• Are model versions pinned and prompts version-controlled?
• Is a post-launch owner named?

Three or more unchecked boxes, and the project is earlier in its life than the demo made it look. That is not a crisis. It is a re-plan.

A pre-production gate

We recommend a formal gate between pilot and launch, with each item evidenced by an artifact rather than an assurance.

Two implementation notes. First, launch in stages: shadow mode (the system runs, humans still do the work, outputs are compared), then a small percentage of traffic, then the rest. Shadow mode is the cheapest source of real-input evaluation data you will ever get. Second, keep the gate lightweight. It is a two-page document and a one-hour meeting, not a quarter-long program.

The pattern underneath

Every item above is a version of the same mistake: treating the demo as the hard part. The model's capability is now the cheap part; it arrived in an API call. The expensive part is everything that turns a capability into a dependable system — and it is expensive in the unglamorous way, which is why it goes unbudgeted.

Teams that ship plan for this from the first week. They build the evaluation set before the second prompt, write acceptance criteria before the first stakeholder demo, and invite security to the kickoff. Their demos are less impressive. Their launches happen.

If you want a second set of eyes on a stalled project, a production-readiness review is a short engagement, and the checklist above is most of what we walk through.