How to Move an AI Prototype Into Production

An AI prototype demonstrates that the approach works on chosen inputs. Production requires that it works on inputs you did not choose, at a cost you can predict, with behaviour you can explain when it goes wrong. Almost all of that gap is infrastructure.

1. Decide the serving path deliberately

Managed model APIs are the correct default: no GPU capacity planning, no model-serving operations, and a clear cost per request. Self-hosting becomes rational when volume makes per-token pricing worse than reserved GPU capacity, when latency requires co-location, or when data residency forbids the API. Model the crossover point rather than deciding on principle.

2. Make retrieval a pipeline, not a script

• Ingestion with source tracking, so you can explain where an answer came from.
• Chunking strategy versioned alongside the embedding model — changing either invalidates the index.
• Scheduled re-embedding and incremental updates as source documents change.
• Index lifecycle: build to a new index, validate, then switch atomically.
• Permission filtering applied at query time, never assumed from the index contents.

3. Evaluation belongs in CI

Prompt and model changes are code changes with no type system behind them. Build a dataset of representative inputs with expected properties, run it on every pull request, and gate merges on regression. Without this, every improvement is a guess and every rollback is a rumour.

4. Control cost before traffic arrives

• Route by task: small models for classification and extraction, large models only where quality demands it.
• Cache aggressively — exact-match, semantic, and prompt-prefix caching where the provider supports it.
• Per-tenant token budgets and rate limits, enforced at the gateway.
• Track cost per request and per tenant as a first-class metric, not a monthly surprise.

5. Instrument what is specific to AI

Standard APM will tell you latency and errors. You also need retrieval hit quality, token usage distribution, refusal and fallback rates, provider error and throttling rates, and output-length distribution. When quality complaints arrive, these are the only signals that let you answer them.

6. Plan for provider failure

Model APIs have outages and rate limits, and models get deprecated on the provider's schedule. Abstract the provider behind an interface, keep a secondary path warm, and make degraded behaviour a designed experience rather than a stack trace.

The sequencing that works

Evaluation first, then observability, then cost control, then serving optimisation. Teams usually do this in reverse — optimising infrastructure for a system whose quality they cannot measure.