From Demo to Production

A working demo is maybe a fifth of the way there. Five gaps separate something that works when you try it from something that works when everyone does.

The demo works. You showed it to someone and they were impressed.

The gap between that and production is wider here than in most software, because the failure modes are unfamiliar: nondeterministic, silent, and often invisible until someone acts on a wrong answer.

Five gaps, in the order they tend to bite.

1 · Measurement

A demo is validated by trying it. Production needs a repeatable answer to “did this change help.”

Without an eval, every prompt change is a guess, and you will not notice when a provider updates a model under a stable version name.

The minimum: twenty saved examples with expected outputs, run on every change. Every production failure gets added. This is not optional infrastructure — it is the thing that makes the rest of the list tractable.

2 · Failure handling

Demos take the happy path. Production takes everything.

Provider outages and rate limits. Exponential backoff with jitter on 429 and 5xx, a circuit breaker so a provider outage does not cascade, and a decision about degraded behavior: fall back to a smaller model, serve a cached response, or fail honestly. Decide deliberately rather than discovering it during an incident.

Malformed output. Schema-constrained generation plus validation after parsing plus one retry. Never assume the shape.

Timeouts. Generations hang. Set them, and handle partial streamed output that stops midway.

Bad input. Empty strings, enormous pastes, other languages, prompt injection attempts. Validate and bound input length before spending a call on it.

3 · Observability

When a user reports a bad answer next week, you need to reconstruct what happened.

Log per request: the full prompt sent (including retrieved context), the raw response, model and version, token counts, latency, and any retries. Sample rather than logging everything if volume demands, but keep all failures.

Retrieved context is the one people omit, and it is the one you need most — for RAG failures, the question is almost always whether the right chunk was retrieved, and you cannot answer it after the fact without the log.

Alert on: error rate, latency percentiles, cost per hour, and refusal or fallback rate. A silent cost spike is the classic unmonitored failure.

Give users a feedback path. A thumbs-down button is the cheapest source of real eval cases you will ever build.

4 · Cost and abuse

A demo serves one considerate user. Production serves everyone, including someone pasting a novel into your summarizer.

Bound input length before the call. This is both a cost control and an availability control.

Rate limit per user, not just globally.

Cap max_tokens so a runaway generation cannot cost unboundedly.

Watch the quadratic. Conversation history is resent every turn, so long sessions cost far more than they feel like they should. Cap history deliberately.

Alert on spend, hourly rather than monthly. The monthly bill is a bad place to learn about a loop.

5 · Safety and trust

Never put API keys in client code. All calls route through your backend. See Your First LLM API Call.

Enforce permissions at retrieval. If RAG indexes documents with differing access levels, filter by the requesting user’s permissions at query time. Otherwise your assistant becomes a way to read things people cannot read directly — a genuinely common and serious mistake.

Treat tool results as untrusted. Anything a tool fetches can contain instruction-shaped text. Keep destructive tools unreachable from untrusted content.

Gate irreversible actions. Sending, deleting, paying, deploying — human confirmation. Prefer reversible operations: draft over send, soft delete over delete.

Set expectations in the interface. Users trust fluent output. Cite sources where you have them, and make it visible that output can be wrong. This is a product decision that reduces real harm.

The shape that works

Successful production systems are narrower than demos suggest.

Scope tightly. One well-defined task beats a general assistant. Easier to evaluate, easier to make reliable.

Prefer workflows to agents where the steps are known. Predictable, cheaper, debuggable. Reach for agents only when the path genuinely cannot be determined in advance.

Keep humans at consequential points. Review before irreversible action, especially early.

Ship to a small group first. Real users find failure modes you will not imagine.

What to remember

• Evals first — without them, no other improvement is measurable.
• Handle rate limits with backoff, malformed output with constrained generation plus validation, and decide degraded behavior before an incident.
• Log the full prompt including retrieved context; alert on error rate, latency, and hourly cost.
• Bound input, rate limit per user, cap output, and watch quadratic history growth.
• Keys server-side, permissions enforced at retrieval, tool results untrusted, irreversible actions gated.
• Narrow scope, workflows over agents where possible, humans at consequential points.

You have reached the end of the path. The roadmap has the full map if you want to revisit a layer.