The 9-Step ML System Design Framework

A repeatable structure for approaching any "design an ML system for X" problem — in an interview or in a real project kickoff.

Click a step. Full depth on each lives in the 9-Step Framework.

Translate a business need into an ML problem.

1. Problem Formulation

Translate a vague business goal ("increase engagement") into a concrete ML problem ("predict the probability a user clicks this post, rank by that probability"). This step is where most real-world ML projects succeed or fail — a well-specified ML problem with a mediocre model beats a vague problem with a great model. Ask: what exactly are we predicting, for whom, and what decision does that prediction drive?

Turn a business request into a trainable objective

The model can only optimize the concrete final specification. Start with a vague business outcome.

2. Metrics

Offline metrics: what you optimize and measure during development — precision/recall, RMSE, ranking metrics like NDCG (see Model Evaluation & Metrics).
Online metrics: what you actually measure once the system is live — click-through rate, retention, revenue, latency. Offline metrics are a proxy for online metrics, and the two don't always move together — a model that improves offline accuracy can still hurt a business metric if it wasn't optimizing for the right proxy.

Offline improvement is a hypothesis

Validate the proxy against real user or business outcomes. Evaluate on a fixed past dataset.

3. Architectural Components (MVP Logic)

Sketch the system end to end before diving into any one piece: where does data come from, what features get computed, what model produces a prediction, how does that prediction reach the end user, and on what latency budget. This high-level sketch is what keeps a design conversation from prematurely diving into model architecture details before the overall shape is agreed on.

4. Data Collection and Preparation

Where does labeled (or implicit) training data come from? User interactions (clicks, purchases) are the most common implicit label source; explicit human labeling is expensive but sometimes necessary. Consider bias in how the data was collected — data collected under an old system's behavior encodes that system's blind spots.

clicked

not clicked

clicked

never shown

not clicked

never shown

Fixed by exploration strategies (deliberately showing some uncertain items) rather than training purely on the old system's exposure pattern.

Items the OLD system never showed (dim) generated zero click data -- not because users wouldn't have liked them, but because they never had a chance. Training on this data alone perpetuates the old system's blind spots.

5. Feature Engineering

What signals actually predict the target? For most production systems this includes user features (history, preferences), item features (content, metadata), and context features (time of day, device, location). See ML Workflow Fundamentals for the general principles.

User features

Item features

Context features

time of day

device

location

Most production systems combine all three -- a prediction usually depends on who, what, and when together.

The situation right now -- signals that change the right answer even for the same user and item.

6. Model Development and Offline Evaluation

Pick a model class appropriate to the problem's data volume, latency budget, and interpretability needs — often starting with a strong, simple baseline (logistic regression, gradient boosting) before justifying anything more complex.

Earn complexity with measured value

Start with a baseline and add complexity only for a proven gap. Fast, debuggable model establishes a reference.

Validate offline using the metrics chosen in step 2, with a train/validation/test split that reflects how the model will actually be used (e.g. splitting by time, not randomly, if the model predicts the future).

Split strategy

■ train■ val■ test

Time-based split: train on the past, validate and test on strictly later periods -- matches how the model will actually be used (predicting forward), so offline evaluation isn't artificially optimistic.

7. Prediction Service

Batch serving: precompute predictions on a schedule (e.g. nightly) — fine when predictions don't need to reflect real-time state.
Online serving: compute predictions on demand, in response to a live request — needed when freshness matters (e.g. ranking a feed based on what just happened).

Latency budget matters enormously here: a search ranking system might have a ~100ms budget for the entire request, which directly constrains model size and feature complexity.

A 100ms budget spans the whole request path

Model inference is only one consumer of latency. Request and response travel time.

8. Online Testing and Model Deployment

A/B testing: route a fraction of traffic to the new model, compare online metrics against the current system, using the hypothesis-testing principles from Probability & Statistics to judge whether an observed improvement is real.
Shadow deployment: run the new model alongside the current one in production without acting on its output, to observe its behavior on real traffic risk-free before it affects users.
Canary rollout: gradually increase the new model's traffic share, watching for regressions at each stage before going to 100%.

Strategy

The axis that matters: does the new model's output actually reach the user, and how much of the traffic.

Canary rollout: a SMALL, growing percentage of traffic gets the new model's real output, watched closely -- ramping toward 100% only if no regressions appear at each stage.

9. Scaling, Monitoring, and Updates

Once live, a model isn't done — it needs monitoring for performance degradation (concept drift, see Model Evaluation & Metrics), a defined retraining cadence, and a rollback plan if a new version underperforms. This is the step most interview answers skip, and the step real production systems live or die by.

A live model is never "done" -- this loop is the step most interview answers skip, and the one real production systems live or die by.

Performance drops below an acceptable threshold, or drift crosses a set boundary.

Next: Case Studies — applying this framework to recommendation systems, search, and GenAI/LLM system design.