Serving & MLOps / LLMOps

A model that only runs in a notebook isn't a production system. This is the tooling layer that makes it one.

Model Serving

vLLM: an inference server purpose-built for LLMs, implementing Paged Attention and continuous batching (see Evaluation & Serving) to serve many concurrent requests efficiently — the de facto standard for self-hosting open LLMs at any meaningful scale. Real discrete-step simulation of exactly why continuous batching wins:

batch capacity4

Static batching (makespan 74)

Continuous batching (makespan 53)

Each dot = one real request's completion time. Continuous batching's dots cluster earlier and more evenly -- no request waits behind a slow neighbor that isn't even in its own batch slot anymore.

Real discrete-step simulation: 24 real requests with real varying output lengths, batch capacity=4. Static batching (short sequences blocked behind the batch's longest one, real makespan=74) vs. continuous batching (a finished slot is refilled immediately, real makespan=53). Real average latency: static=47.3 steps, continuous=29.2 steps -- 38% real improvement, the actual mechanism behind vLLM's throughput advantage, not just the claim.

TorchServe: general-purpose model serving for PyTorch models, handling request batching, versioning, and scaling for traditional (non-LLM) deep learning models.
Triton Inference Server: a high-performance serving platform supporting multiple frameworks (PyTorch, TensorFlow, ONNX) and multiple models on shared hardware, commonly used when a system serves a mix of model types, not just LLMs.

Containerization & Deployment

Docker packages a model plus its exact runtime environment (dependencies, system libraries) into a single portable image — solving the "it works on my machine" problem and making deployment to any cloud or on-prem environment consistent. Nearly every production ML serving setup runs inside containers, orchestrated by something like Kubernetes at scale.

CI/CD for ML

Traditional software CI/CD (test, build, deploy on every code change) extends to ML with extra pieces specific to models: automated retraining pipelines (trigger on a schedule or when enough new data has accumulated), automated evaluation gates (a new model version must beat the current one on held-out metrics before it's allowed to deploy — connecting back to Model Evaluation & Metrics), and automated rollback if a newly deployed model's live metrics regress.

Observability for LLM Applications

Unlike traditional software, LLM application behavior isn't fully deterministic or easy to unit-test in the usual sense — so observability focuses on:

Tracing: logging the full sequence of prompts, tool calls, and responses for a given request, so you can debug why an agent or RAG pipeline produced a particular output.
Cost tracking: monitoring token usage and cost per request/user, since LLM API costs scale directly with usage in a way traditional compute costs don't as visibly.
Quality monitoring in production: sampling live outputs for automated or human quality review (see LLM-as-judge) to catch regressions that offline evaluation didn't anticipate.

Semantic Caching

Exact-match caching only helps when the same string comes in twice — useless for two users asking "what's your refund policy" and "how do refunds work." Semantic caching embeds each incoming request and checks vector similarity against previously-cached queries, so semantically equivalent-but-differently-worded requests can still hit the cache and skip a real LLM call entirely. GPTCache is the reference open-source implementation — a modular pipeline (embedding model → vector store → similarity evaluator → cache storage) that plugs into an existing application rather than requiring infrastructure changes.

It's not a free win, and hit rates vary enormously by workload — template-heavy agent inner loops (the same sub-task repeated with minor variation) see 40-70% hit rates, while long-tail conversational agents see closer to 10-25%, since fewer requests are actually similar enough to reuse. It also has a real, documented failure mode: pure embedding-similarity matching can wrongly cache-hit two queries that are textually similar but carry different intent, or miss two queries that are worded completely differently but mean the same thing — current work on this ("Why Agent Caching Fails and How to Fix It") proposes canonicalizing a query's structured intent via an LLM before the similarity check, rather than trusting embedding similarity alone. Worth treating as a real correctness lever to tune (similarity threshold, what counts as "equivalent"), not a drop-in optimization with no downside.

Frameworks section complete. Next: Interview Prep — putting everything from the previous sections into practice under interview conditions.