PoC to Production

The idea in one minute#

A proof of concept (PoC) shows that something can work once, on chosen examples, for a friendly audience. Production means it works every day, on whatever arrives, for people who did not ask for it, inside the organisation’s security, cost and support rules. The distance between the two is where most enterprise AI projects end.

They rarely end because the model was not good enough. They end because the PoC was built to impress rather than to learn, had no success criterion, used data that production cannot reach, and left security, operations and the users themselves for later.

The cure is to treat delivery as four stages with a gate between each — prototype, pilot, limited release, full release — and to design the gates before the first line of code. Each stage exists to remove one kind of risk. A stage that removes no risk is a demo.

A picture#

flowchart LR
  P0[":i-flask-conical: <b>Prototype</b><br/><small>2–4 weeks · can it work at all?</small>"] -->|"gate: feasible on real samples"| P1[":i-users: <b>Pilot</b><br/><small>8–12 weeks · real users, real data</small>"]
  P1 -->|"gate: eval + business metric + adoption"| P2[":i-split: <b>Limited release</b><br/><small>10–50% of traffic</small>"]
  P2 -->|"gate: holds at scale, cost as modelled"| P3[":i-check: <b>Full release</b><br/><small>operated, monitored, owned</small>"]
  P0 -.->|"fails"| K[":i-archive: <b>Stop, with findings</b><br/><small>a cheap, useful outcome</small>"]
  P1 -.->|"fails"| K
  SEC[":i-shield-check: <b>Security, legal, operations</b><br/><small>engaged from the prototype</small>"] -.-> P0
  SEC -.-> P1
  SEC -.-> P2
  class P0 neutral
  class P1 compute
  class P2 queue
  class P3 io
  class K memory
  class SEC warn

How it really works#

Why pilots stall#

The same causes appear in every study and every post-mortem. Each has a design-time fix.

The MIT Project NANDA study put the point sharply: the organisations that got value treated adoption as a workflow and learning problem, while those that stalled treated it as a technology purchase.

The four stages#

Stopping after a prototype that fails its gate is a success of the process. It cost three weeks instead of nine months.

Design the gates first#

Write all three gates before the prototype begins. They come straight from the requirements page.

GATE 1 — prototype → pilot
  □ ≥ 90% field accuracy on a random sample of 50 real claims
  □ Measured token cost within 2× of the estimate
  □ Live data access agreed in writing with the document platform owner
  □ Security and privacy have reviewed the design; no blocking finding
  □ Sponsor confirms the pilot team and the baseline measurement

GATE 2 — pilot → limited release
  □ Evaluation gate passed on the full set (see the evaluation plan)
  □ Reading time per claim ≤ 6 minutes, measured, for the pilot group
  □ ≥ 70% of pilot handlers use it on ≥ 80% of their claims in the final four weeks
  □ No severity-1 incident; all high-risk review findings closed
  □ Unit cost at pilot volume ≤ $0.08 median

GATE 3 — limited release → full release
  □ Metrics hold at 40% of volume for four weeks
  □ Peak-hour load test passed against granted quota
  □ Runbooks exercised: provider outage, quality drop, rollback
  □ Product owner, run team and budget confirmed for 12 months

Each line is checkable by someone who was not in the project. That is the test of a gate.

A pilot is an experiment#

Run it like one.

• Measure the baseline first. Reading time before the pilot, for the same people, on the same kind of claims. Without it the result is an opinion.
• Have a comparison. A group not using the tool, or the same group before and after.
• Fix the duration. Long enough to pass the novelty period — eight to twelve weeks. Usage in week one tells you nothing; usage in week eight tells you nearly everything.
• Instrument adoption. Who used it, how often, where they abandoned it, what they corrected.
• Collect the failures. Every correction and complaint is a future test case.
• Report against the gate, including the lines that failed.

Releasing safely#

Models, prompts and retrieval indexes are released with the same care as code — and independently of it.

Three things are versioned and released together: the prompt, the model version, and the retrieval index or tool set. A change to any one can change behaviour; the evaluation gate applies to each.

What “operated” means#

A system is in production when someone can answer these at three in the morning.

The telemetry behind these answers is covered in Observability Engineering.

The part that is not technical#

Technology is the smaller half of a rollout. Budget time and money for:

• Training — not how to click, but when to trust and when to check.
• Process change — if handlers save twenty minutes, what does the process now expect of them?
• Honest communication — what the tool does, what it does not, and what it means for people’s jobs. Silence is filled with the worst assumption.
• Champions — two or three respected users who helped design it and will answer their colleagues’ questions.
• A visible feedback route — and visible fixes. Nothing builds adoption like a complaint resolved within the week.

The architect’s role, stage by stage#

Remember this#

• A PoC proves it can work once. Production means every day, on anything, within the rules.
• Four stages, each removing one risk: feasibility, value and fit, scale, sustainability.
• Write all the gates before the prototype. Every line must be checkable by an outsider.
• A pilot is an experiment: baseline, comparison, fixed duration, adoption data.
• Involve security, data owners, operations and users from the prototype — not at launch.

Try it#

1. Take a PoC you have seen stall. Which rows of the “why pilots stall” table applied? Which one fix would have mattered most?
2. Write gate 1 and gate 2 for a use case of your own. Give them to a colleague and ask whether they could check each line without you.
3. Draft the three-in-the-morning table for the same system. Which rows have no owner yet?

Check yourself#

1. What risk does each of the four stages remove?
2. Why is a prototype that fails its gate a good outcome?
3. Which three things are versioned and released together in an AI system?
4. Why must a pilot run longer than a few weeks?

Sources#

• MIT Project NANDA: The GenAI Divide — State of AI in Business 2025 (press coverage of a preliminary report)
• Microsoft Cloud Adoption Framework: AI adoption — Strategy, Plan, Ready, Govern, Secure, Manage
• AWS Prescriptive Guidance: Operationalizing agentic AI