ML System Design Case Studies

Applying the 9-step framework to the problem types that come up constantly, both in interviews and in real systems.

Learning From a Real Problem, Not a Topic List

Nobody sets out to "learn Vector Databases" — they set out to build something, hit a wall a plain keyword search can't get past, and that's when a vector database becomes the obvious next thing to learn, motivated by a real constraint instead of arriving cold. The case studies below are built that way on purpose: each one narrates a real system end to end, and every technology shows up exactly where the problem actually demands it, linking out to this site's real deep-dive page for that piece rather than re-explaining it here. This page's job is motivation and sequencing; the linked pages own the actual depth.

Case Study: A RAG System Over 100M Enterprise Documents

The requirement sounds simple — "let people ask questions and get answers grounded in our own documents" — until the real numbers show up: 100 million documents, sub-second latency expectations, and answers that have to be right, not just plausible-sounding, because a wrong answer in an enterprise setting has real consequences. Walk it in the order the constraints actually force:

Each step, in the order it's actually forced, with the real page that owns the depth:

1. Chunking — first wall: you can't hand 100M documents to a model at once, and even one long document usually blows the context window. See Chunking Strategies for how to split documents so a chunk is retrievable on its own without losing the context that made it meaningful in the first place — the actual hard part, not just "split every 500 words."
2. Embeddings — chunks need to be searchable by meaning, not just keyword-matched: a question phrased differently than the document should still find it. See Choosing an Embedding Model for the real tradeoffs (dimensionality, domain fit, cost) in picking the model that turns each chunk into a vector.
3. Vector DB — 100M embedded chunks is too many to brute-force compare against on every query; you need a real index, not a linear scan. See Vector Databases for how approximate nearest-neighbor indexing makes that search fast at this scale, and what accuracy it trades away to get there.
4. Hybrid Search — pure embedding search misses things a keyword search would catch instantly (an exact product code, a specific error string) — semantic similarity isn't the same as relevance for every query. See Dense vs. Sparse Retrieval, and BM25 for combining both instead of picking one.
5. Reranking — retrieval has to be fast, so it over-fetches (the top 50 plausible chunks, cheaply) — but "plausible" and "actually the best" aren't the same list. See Re-ranking with Cross-Encoders for the expensive-but-accurate second pass that reorders those 50 down to the handful that actually go in the prompt.
6. Caching — at real query volume, a meaningful fraction of retrieval + generation work is redundant: the same system prompt, similar questions, repeated sub-queries. See KV Cache & Prefix Caching for reusing computation across requests instead of redoing it from scratch every time.
7. Eval — "it gives an answer" isn't the same as "it gives a right, grounded answer," and you won't know the difference without measuring it. See Evaluating RAG for faithfulness/relevance metrics that catch hallucination and irrelevant retrieval specifically, not just generic answer quality.
8. Observability — when an answer is wrong in production, you need to know whether retrieval found the wrong chunks or generation misread the right ones — two completely different bugs with the same symptom. See Observability for tracing a request through every stage of the pipeline, not just logging the final output.
9. Security — enterprise documents are exactly the kind of untrusted content an attacker can plant a prompt injection inside, and a poisoned document can corrupt what the whole system retrieves for everyone. See AI Red Teaming for testing specifically against retrieval/RAG-poisoning attacks, not just the model's own prompt-injection surface.
10. Scaling — enterprise traffic doesn't arrive evenly, and embedding + reranking inference is its own real compute cost at 100M-document scale, separate from the LLM call itself. See Embedding & Reranker Inference for Production RAG for serving that piece of the pipeline at real load, not just the generation step.

Every one of those ten links is depth this site already has — this page's contribution is the order they actually get discovered in, and the specific reason each one becomes necessary, which is a different (and more durable) kind of understanding than reading the same ten pages as an unordered list of "things RAG systems use."

Case Study: Real-Time Fraud Detection at 100K Events/Sec

Different shape of problem entirely: not "search accurately," but "decide, correctly, on every single transaction, in milliseconds, at a sustained 100,000 events per second, where fraud is rare and getting it wrong in either direction is expensive." Walk the same way:

1. Event Stream — 100K events/sec arriving continuously isn't a batch job; nothing about "load a CSV and run inference" survives contact with a real transaction stream. See Message Queues & Async Processing for the producer/queue/consumer shape that decouples "a transaction happened" from "something processed it," and Kafka, Celery, Redis, SQS: Picking One for why Kafka specifically is the real answer at this throughput.
2. Features — a model needs real features per transaction (spending velocity, merchant history, device fingerprint) computed identically online and offline; get that wrong and the model trained on one version of a feature scores on a subtly different one in production. See The Training-Serving Skew Problem and Feature Stores for the shared pipeline that makes training and serving compute features the exact same way.
3. Imbalance — fraud is rare, often well under 1% of transactions, so a model can hit 99%+ accuracy by never flagging anything, which is useless. See Handling Messy Data and Sampling Strategies for resampling and class-weighting approaches that keep a rare-class problem learnable.
4. Serving — a model sitting in a notebook doesn't score a live transaction in milliseconds; it needs a real, low-latency serving path in the critical path of every single payment. See APIs & Model Serving for the real online-inference shape this requires, distinct from the batch/offline serving a slower problem could get away with.
5. Threshold — the model outputs a probability, not a decision, and a missed fraud case costs far more than a false alarm, so the right cutoff is almost never 0.5. See Threshold Optimization for choosing the cutoff deliberately from the real cost asymmetry, instead of defaulting to the number that looks best on a leaderboard metric.
6. Monitoring — fraud patterns actively adapt to whatever the current model catches; a model accurate at launch degrades on its own as fraudsters route around it, with no code change on your side at all. See Monitoring & Drift Detection for telling "the world changed" apart from "the pipeline broke," and catching it before it shows up as a business loss — and ml-drift-monitor for exactly this step built end to end around a real fraud model, with the detector's own false-positive rate and detection power empirically measured against known, injected drift rather than assumed correct.
7. Rollout — a new fraud model can't just replace the old one and hope; a regression here means real financial loss, silently, until someone notices. See Statistical Significance of Improvements for confirming a new model is genuinely better before it takes over 100% of live traffic, not just better on one offline split.

Notice the shape difference from the RAG case study even though the underlying framework (the 9-step design process) is identical: a search problem's constraints are mostly about retrieval quality at scale; a fraud problem's constraints are mostly about streaming infrastructure and decision cost — same process, genuinely different technology stack, because the technology stack is downstream of the problem, not chosen first.

Case Study: A Support Agent That Can Actually Take Actions

A third shape again: not "retrieve accurately" or "decide fast on a stream," but "let a model do things — look up an order, issue a refund, escalate a case — safely, for 50,000 conversations a day, without either being uselessly restricted or dangerously unrestricted." This is where the constraints stop being about data or infrastructure and start being about trust:

1. Tool Use — answering from training data isn't enough; the agent needs to actually look up a real order, not describe what looking one up would involve. See Tool Use / Function Calling for how a model actually gets access to real systems instead of just talking about them.
2. The Loop — one tool call rarely resolves a real support request; the agent needs to reason about what the lookup returned and decide what to do next, possibly several times. See ReAct for the reason-act-observe cycle this requires, and How We Got Here for why a single prompt was never going to be enough in the first place.
3. Memory — a real support conversation spans multiple turns; forgetting what the customer already said two messages ago is a real, visible failure mode, not a hypothetical one. See Agent Memory for short-term vs. long-term memory and what belongs in each.
4. Reversibility — looking up an order and issuing a refund are not the same risk, and treating every action as equally sensitive (or equally safe) is wrong in both directions. See Gate on Reversibility, Not Confidence for sorting actions by actual blast radius instead of a single uniform approval policy.
5. Excessive Agency — an agent with real refund access can be talked (by a confused customer, or a deliberately adversarial one) into using it for a request that never needed it. See Excessive Agency for the real OWASP-named failure mode this is, and why scoping what an agent can do matters as much as what it's told to do.
6. Guardrails — a well-written prompt is not a control; it's a strong suggestion the model usually follows, which isn't the same guarantee a real system needs for a refund path. See Guardrails for the systematic constraints that hold even when the prompt alone wouldn't.
7. Human-in-the-Loop — full autonomy on a real refund is a real financial risk; requiring manual review of every single interaction defeats the entire point of building the agent. See Human-in-the-Loop for approval gates placed specifically at the high-consequence steps, not sprinkled everywhere.
8. Durability — a real conversation can be interrupted, retried, or resumed hours later; losing all context because the process restarted is a real, visible bug, not an edge case. See Durable Execution for checkpointing a long-running or interrupted agent task properly.
9. Observability — when an agent does the wrong thing, you need to see the actual reasoning and tool calls that led there, not just the final message the customer saw. See Observability for tracing what actually happened, not just logging outcomes.
10. Evaluation — "the response sounds right" is not the same claim as "the agent did the right thing," and conflating them hides real failures behind plausible-sounding text. See Evaluation for checking agent behavior specifically, not just output fluency.

Case Study: Personalized Recommendations for a 50-Million-Item Catalog

A fourth shape: not "retrieve accurately," "decide fast on a stream," or "act safely," but "rank the right handful of items out of tens of millions, differently for every single user, where the system's own past choices quietly shape the data it gets to learn from next." Walk it the same way:

1. Candidate Generation — the first wall: scoring all 50 million items for every user request in real time is computationally impossible, so the system has to cheaply narrow the catalog down to a few hundred plausible candidates before anything expensive happens. See The Two-Stage Production Pipeline for why this split (cheap-and-broad, then expensive-and-narrow) is the standard shape almost every large-scale recommender takes.

2. Embeddings — candidate generation needs a cheap, precomputed way to represent user preference and item similarity numerically, rather than comparing raw interaction history at request time. See Matrix Factorization for the classical latent-factor version of this, and Embeddings and the Two-Tower Architecture for the modern, feature-rich neural version that also partially addresses cold start below.
3. ANN Retrieval — comparing a user's embedding against 50 million item embeddings one at a time, per request, is still too slow even once both sides are just vectors. See Indexing Algorithms for the approximate nearest-neighbor index (the same HNSW-style structure a RAG system's vector DB uses) that makes this lookup fast at real catalog scale.
4. Ranking — the few hundred retrieved candidates aren't yet ordered by actual predicted preference; approximate embedding similarity got you a plausible shortlist, not a precise one. See Three Formulations for pointwise/pairwise/listwise ranking losses, and CTR Prediction for the concrete binary-classification framing most production ranking-stage models actually train against.
5. Cold Start — brand-new users and items have no interaction history for matrix factorization or embeddings to have learned anything from yet, and the two-stage pipeline above quietly assumes that history exists. Addressed with content-based fallbacks (rank by item attributes instead of behavior) and deliberate exploration (show some uncertain-but-promising items specifically to gather signal, not because the model is already confident in them):

6. Feedback Loop Bias — the model's own past recommendations shape what a user even gets the chance to click on, which becomes tomorrow's training data — quietly reinforcing whatever the model already believed instead of correcting it. See The Explore/Exploit Problem, Stripped to Its Purest Form for why pure exploitation (always show the current-best guess) actively poisons future training data, and Contextual Bandits for the framework that treats "which item to show" as a decision made under genuine uncertainty, not a settled fact.
7. Training-Serving Skew — features like recency, session activity, and inventory state computed differently in the training pipeline than at request time silently corrupt what the model actually learned, the same failure mode a streaming fraud system hits for the same underlying reason. See The Training-Serving Skew Problem and Feature Stores for the shared pipeline that keeps both sides honest.
8. Evaluation — an offline ranking metric improving doesn't prove real engagement or revenue actually went up; the two can and do diverge. See Ranking Evaluation Metrics for NDCG/MRR as the offline signal, and Statistical Significance of Improvements for confirming an online lift is real before rolling a new ranker out to 100% of traffic.
9. Scaling — every one of millions of daily requests needs its own personalized, low-latency answer, not a shared batch result computed once and served to everyone. See Batch, Online, and Streaming Inference for why this specific problem shape forces the online-inference path, not the batch one a less personalized system could get away with.

Notice the shape difference again: RAG's constraints were about retrieval quality, fraud's about streaming infrastructure and decision cost, the agent's about trust and reversibility — this one's constraints are almost entirely about a system that trains on the consequences of its own past decisions, which none of the other three problems have to contend with at all.

Four case studies, four genuinely different technology stacks — retrieval-heavy, streaming-and-decision-heavy, trust-and-control-heavy, feedback-loop-heavy — from applying the exact same 9-step framework to four differently-shaped problems. That's the actual claim this page is making: the framework is constant, the stack is downstream of the problem.

Search & Ranking

Similar two-stage shape: retrieval (find documents/items that could plausibly match a query — keyword-based, or embedding-based, or both — see Hybrid Search) followed by learning-to-rank, a model trained specifically to order the retrieved set by relevance, typically using pairwise or listwise ranking losses rather than plain classification/regression.

News Feed / Ads Ranking

A specialized case of ranking, usually optimizing a blended objective (engagement and revenue, for ads) with heavy weight on real-time signals (what a user just did in this session) and strict latency constraints, since feed ranking happens on every single scroll.

NLP and Computer Vision Systems at Scale

Follow the same 9-step shape, with domain-specific model choices — Transformers (see Deep Learning) for NLP, CNNs or ViTs for vision — but the system design concerns (latency budgets, monitoring, retraining cadence, A/B testing) are largely the same regardless of domain.

GenAI / LLM System Design (2026-era)

The newest and increasingly most-asked case study category:

• RAG pipeline design: applying the 9-step framework to a retrieval-augmented system — see RAG for the pipeline details, but frame it explicitly through metrics (offline: faithfulness/relevance; online: user satisfaction, task completion) and a serving architecture (embedding service, vector DB, LLM inference, all with their own latency budgets).
• Agent system design: designing a multi-step agentic workflow (see Agents) as a production system — including cost controls (agents can call tools and other LLMs repeatedly, so cost isn't a single inference call anymore), reliability (retries, fallbacks when a step fails), and observability (tracing what an agent actually did, since its behavior isn't fully deterministic).
• LLM serving infrastructure: applying steps 7-9 (prediction service, deployment, monitoring) to LLM-specific serving concerns — batching, KV cache management, and cost-per-token monitoring (see Evaluation & Serving).

Common Problems & SOTA Solutions

• Training-serving skew (features computed differently in training vs. production) → shared feature pipelines / feature stores, so the exact same code computes a feature both times
• Cold start → content-based fallback, exploration strategies, popularity-based defaults until enough signal accumulates
• Feedback loops biasing the model (a ranking model's own past recommendations shape what users click, reinforcing its own biases) → randomized exploration traffic, counterfactual/off-policy evaluation
• Model staleness / concept drift → monitoring dashboards tracking live metrics vs. offline expectations, scheduled retraining, online learning for fast-moving domains
• Scaling inference to millions of requests → caching frequent predictions, batching, model distillation to a smaller serving model, horizontal scaling behind a load balancer

ML System Design section complete. Next: Databases — the storage layer every one of these systems depends on.