Production Reliability

Everything covered so far in MLOps gets a model trained, deployed, and monitored. This page is the difference between "the demo works" and "the system survives real traffic, real failures, and real edge cases without falling over" — AI demo engineering vs. AI production engineering, as distinct disciplines.

Retries and Timeouts

• Timeouts: every external call (a downstream API, a database query, an LLM provider call) needs an explicit maximum wait time — without one, a single slow dependency can hang a request indefinitely, and under load, enough hung requests exhaust the server's available connections/threads, turning one slow dependency into a full outage.

• Retries: transient failures (a momentary network blip, a brief downstream overload) often succeed on a second attempt — retrying automatically, rather than failing the whole request on the first error, meaningfully improves reliability for exactly this class of failure. Retries should have a capped maximum count (an unlimited retry loop against a truly broken dependency just adds load to an already-struggling system) and backoff (waiting progressively longer between attempts, often with a small random jitter added to prevent many clients from retrying in synchronized bursts that re-overload the dependency the moment it recovers).

Both timeout and capped, jittered retry as real Python (tenacity is the standard library for exactly this):

from tenacity import retry, stop_after_attempt, wait_random_exponential
import httpx

@retry(stop=stop_after_attempt(3), wait=wait_random_exponential(multiplier=0.5, max=8))
def call_llm_provider(prompt: str) -> str:
    response = httpx.post(LLM_API_URL, json={"prompt": prompt}, timeout=5.0)  # explicit timeout, every call
    response.raise_for_status()
    return response.json()["text"]

Circuit Breakers

A circuit breaker tracks a dependency's recent failure rate and, once it crosses a threshold, stops sending requests to that dependency entirely for a cooldown period — rather than continuing to retry against something that's clearly down, which wastes resources and adds latency to every request without any chance of success. The pattern name is deliberate: like an electrical circuit breaker, it "trips" open on sustained failure, and periodically allows a small number of test requests through to check whether the dependency has recovered before fully "closing" again. This turns a downstream outage into a fast, predictable failure for callers instead of every request hanging until its own timeout expires.

import pybreaker

llm_breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=30)

@llm_breaker
def call_llm_provider(prompt: str) -> str:
    return httpx.post(LLM_API_URL, json={"prompt": prompt}, timeout=5.0).json()["text"]

# after 5 consecutive failures, further calls raise CircuitBreakerError immediately
# (no network round-trip, no waiting on a timeout) until the 30s reset_timeout elapses

Fallback Models and Graceful Degradation

• Fallback models: if a primary model (or the primary inference engine serving it) is unavailable or exceeding its latency budget, fall back to a smaller, faster, or simpler model rather than failing the request outright — a degraded response is usually better than no response, and this is exactly why multi-LoRA serving and model routing architectures often keep a cheap fallback path alongside the primary model.
• Graceful degradation: more generally, a system should fail by doing less, not by failing completely — if a RAG system's retrieval step is down, answering from the LLM's own knowledge (clearly caveated as ungrounded) is more graceful than returning an error; if personalization data is unavailable, falling back to generic (non-personalized) recommendations keeps the core feature working rather than breaking entirely. Identify, in advance, which parts of a system are "nice to have" versus "must work," and design the failure behavior for each accordingly — rather than treating every dependency as equally critical by default.

Rate Limits and Backpressure

• Rate limits: cap how many requests a given client (or the system overall) can make in a given time window — protects the system (and, for a paid upstream API, your budget) from a single misbehaving client or an unexpected traffic spike overwhelming shared capacity.
• Backpressure: when incoming request volume exceeds what a system can currently handle, backpressure means explicitly signaling "slow down" back to callers (rejecting or queuing new requests) rather than silently accepting more work than can actually be processed, which just delays every request further and can eventually crash the service entirely. A system with no backpressure mechanism degrades catastrophically under overload; a system with backpressure degrades predictably (some requests rejected or queued, the rest served normally) — the difference between a bad afternoon and a full outage.
• Queues: decouple request arrival from request processing — incoming requests land in a queue, and workers process them at a sustainable rate, absorbing traffic bursts smoothly rather than requiring capacity provisioned for worst-case instantaneous load. Directly relevant to GPU-backed LLM serving specifically, where provisioning for peak load is expensive enough that smoothing bursts through a queue is often more cost-effective than always having peak capacity on standby.

Idempotency and Distributed Locks

• Idempotency: an operation is idempotent if performing it multiple times has the same effect as performing it once — critical for anything triggered by a retry (above): if a "charge the customer" or "send this notification" action isn't idempotent, a retried request (after a timeout where the first attempt actually did succeed, just slowly) can duplicate the action. The standard fix is an idempotency key — a unique identifier per logical operation that the receiving system checks against, rejecting a duplicate request carrying an already-processed key.

@app.post("/v1/charge")
def charge(req: ChargeRequest, idempotency_key: str = Header(...)):
    if (cached := redis.get(f"idempotency:{idempotency_key}")):
        return json.loads(cached)   # already processed -- return the original result, don't charge again

    result = process_charge(req)
    redis.set(f"idempotency:{idempotency_key}", json.dumps(result), ex=86400)
    return result

• Distributed locks: when multiple workers/processes might operate on the same resource concurrently (two training jobs writing to the same checkpoint location, two workers processing the same queued item), a distributed lock ensures only one actually proceeds at a time — implemented via a coordination service (the same kind of majority-consensus mechanism as CS Fundamentals — Consensus) rather than assuming single-process, single-machine execution.

Disaster Recovery and Rollback

• Disaster recovery: a documented, tested plan for restoring service after a major failure (a region-wide cloud outage, a corrupted database) — backups alone aren't a disaster recovery plan until someone has actually verified they can be restored successfully within an acceptable time window; an untested backup is a hope, not a plan.
• Rollback: the ability to quickly revert to a previously-known-good model/code version when a new deployment causes problems — this is exactly what Deployment Strategies' blue-green pattern provides directly (the old environment stays live, ready to receive traffic back instantly) and what canary deployment enables at a smaller blast radius (only a fraction of traffic needs rolling back). A deployment process with no fast rollback path turns every deploy into a higher-stakes, harder-to-reverse decision than it needs to be.

Next: AI Cost Engineering — reliability and cost are frequently in tension (more redundancy costs more), which is exactly why they're covered back to back.