Reliability Patterns

Providers rate-limit, time out, and return malformed output. The patterns that keep a system up when its dependency is unreliable.

A model API is a network dependency with variable latency, rate limits, occasional outages, and nondeterministic output. Every pattern here is standard distributed-systems practice, applied to a dependency that also returns unpredictable content.

Retry correctly

Retry: 429 rate limits, 5xx server errors, timeouts, connection failures.

Do not retry: 400 malformed requests, 401 auth failures, 403 permission errors, context-length errors. The request itself is wrong; repeating it wastes time and quota.

Use exponential backoff with jitter. Double the wait each attempt, and add randomness so concurrent clients do not retry in lockstep and re-create the spike that caused the rate limit. Respect a retry-after header when one is present.

Cap attempts — three or four. Beyond that you are adding latency to a request that has already failed.

Most SDKs implement this. Confirm rather than assume, and check what they consider retryable.

Circuit breakers

Retrying into a sustained outage makes things worse: every request burns its full retry budget before failing, so latency climbs and threads pile up.

A circuit breaker tracks the failure rate and, past a threshold, fails immediately without calling the provider. After a cooling period it lets a trial request through and closes if that succeeds.

This converts a slow cascading failure into a fast clean one, which is almost always the better outcome.

Decide degraded behaviour in advance

When the model is unavailable, something must happen. Choose it deliberately rather than discovering it during an incident.

Fall back to another provider. Strongest option, and it requires prompts that work on both — worth testing periodically, since a fallback nobody exercises is a fallback that does not work.

Fall back to a smaller model. Same provider, different tier. Degraded quality beats no response.

Serve a cached response for repeated questions.

Fall back to non-AI behaviour. Keyword search instead of semantic search, a template instead of generated text. Frequently the most robust choice.

Fail honestly. Sometimes correct. Say the feature is unavailable rather than returning something wrong.

The wrong answer is silently returning degraded output while presenting it as normal.

Handle malformed output as expected

Non-conforming output is routine, not exceptional. See Getting JSON Out of an LLM.

The layered approach: use schema-constrained generation where the provider offers it, validate after parsing regardless, strip code fences before parsing, and retry once with the validation error included in the prompt. Most failures resolve on the retry.

Watch for truncation specifically. An object cut off mid-generation means your output limit is too low, and retrying will not fix it — that is a configuration bug wearing a parse error’s clothing.

Timeouts and idempotency

Set timeouts explicitly. Generation takes real time and occasionally hangs. Without a timeout, a hung request holds resources indefinitely.

Make retries idempotent. If a call triggers a side effect — sending, charging, writing — a timeout leaves you unsure whether it completed. Use idempotency keys, or write the side effect through a path that tolerates duplicates.

This matters most for agents, where retried tool calls can repeat real actions.

Isolate the failure

Bulkheads. Separate connection pools and concurrency limits per feature, so a slow summarization endpoint cannot exhaust the capacity your critical path needs.

Queues for non-interactive work. Anything nobody is waiting on belongs in a queue with its own retry semantics — and batch endpoints are cheaper for exactly this traffic.

Cap agent budgets. Iterations, tokens, and wall-clock time. An uncapped loop is an unbounded failure.

What to monitor

Error rate by type, retry rate, circuit breaker state, latency percentiles, and validation failure rate. A rising retry rate is the leading indicator of trouble — see Observability for LLM Systems.

What to remember

• Retry 429s, 5xx, and timeouts with exponential backoff and jitter; never retry 400s or 401s.
• Circuit breakers turn slow cascading failures into fast clean ones.
• Choose degraded behaviour before an incident: another provider, a smaller model, a cache, non-AI fallback, or honest failure.
• Treat malformed output as routine: constrain, validate, strip fences, retry once with the error.
• Set timeouts, make retries idempotent — critical when tool calls have side effects.
• Isolate with bulkheads, queue non-interactive work, and cap agent budgets.

Next: Prompt Versioning