Engineering Foundations for ML

The part of MLOps that gets skipped in most ML courses, and is the actual difference between a notebook and a system someone else can run, debug, and trust at 3am.

Why "It Works On My Machine" Isn't Good Enough

A model in a Jupyter notebook has implicit dependencies everywhere: cell execution order, global variables set three cells up, a specific package version installed by hand. None of that survives contact with a second person, a second machine, or six months of time passing. Everything below exists to remove that implicitness.

Clean Code & Architecture

• SOLID principles, applied to ML code specifically: a Model class shouldn't also handle data loading and API serialization (Single Responsibility) — split them so any one piece can change without breaking the others.

• Modular architecture: separate data loading, preprocessing, model definition, training loop, and evaluation into distinct modules — the same separation of concerns that makes any software maintainable, just applied to a domain (ML) that often skips it under deadline pressure.
• Configuration management: hyperparameters, file paths, and environment-specific values belong in config files (YAML/JSON) or environment variables — never hardcoded, so the same code runs in dev/staging/production by changing config, not code.

• Dependency management: pin exact versions (pyproject.toml + a lockfile via uv or poetry) — an ML pipeline that silently picks up a newer NumPy/PyTorch and produces different numbers is a debugging nightmare that pinned dependencies prevent outright.

[project]
name = "fraud-model"
dependencies = [
    "torch==2.4.0",
    "numpy==1.26.4",
    "scikit-learn==1.5.1",
]

uv sync             # installs exactly what's in uv.lock -- no version resolution surprises
uv lock --upgrade   # a deliberate, reviewable step to bump versions, not an implicit side effect of installing

• API design: if your model is served behind an API (see APIs & Model Serving), the contract (request/response schema, error format, versioning) is a design decision made deliberately, not whatever FastAPI happens to auto-generate from your first draft.

Testing an ML System

Testing ML code is genuinely harder than testing normal software, because "correct" isn't just "matches an exact expected value" — a model's output is expected to vary. The test types below cover different layers of that problem:

• Unit tests: test individual functions in isolation — a data-cleaning function, a feature-computation function, a custom loss function's math.
• Integration tests: test that pieces work together — does the full preprocessing pipeline correctly feed the model's expected input shape?
• End-to-end tests: run the entire pipeline (data in, prediction out) on a small fixture dataset, checking it completes without error and produces reasonable output.
• Regression tests: does a code change accidentally degrade a metric that used to pass? Run the model against a fixed benchmark dataset and assert performance hasn't dropped below a threshold.
• Data validation tests: assert incoming data matches expected schema, ranges, and types before it reaches the model — see Data Engineering & Versioning for the pipeline stage this belongs to.
• Model tests: beyond accuracy — does the model handle an empty input gracefully? Does it produce the same output for the same input (determinism, where expected)? Does a known edge case (e.g. a specific adversarial input) behave as expected?
• API tests: does the serving endpoint return the correct schema, correct status codes, and handle malformed requests without crashing?
• Load tests: does the serving endpoint hold up under realistic (or peak) concurrent request volume, and what's the latency distribution under load?

Tools

• pytest: the standard Python test runner — fixtures, parametrized tests, and plugins cover nearly everything above.
• unittest: Python's built-in test framework — less ergonomic than pytest, but zero extra dependencies, and you'll see it in older codebases.
• mock: replace expensive or external dependencies (a real API call, a real database) with a fake stand-in during tests, so tests run fast and don't depend on external services being up.
• Locust: a load-testing tool — define realistic user behavior in Python, and it simulates many concurrent users hitting your API, reporting latency/throughput/error rate under load.
• Postman: manual and scripted API testing/exploration — useful for exploring and documenting an API's actual behavior before automating tests against it.

A real parametrized unit test, and a Locust load test for the serving endpoint from APIs & Model Serving:

import pytest
from features import clean_transaction_amount

@pytest.mark.parametrize("raw, expected", [
    (" $1,200.50 ", 1200.50),
    ("0", 0.0),
    ("-42", None),   # negative amounts are invalid -- should be dropped, not coerced
])
def test_clean_transaction_amount(raw, expected):
    assert clean_transaction_amount(raw) == expected

from locust import HttpUser, task, between

class ModelApiUser(HttpUser):
    wait_time = between(0.1, 0.5)

    @task
    def predict(self):
        self.client.post("/v1/predict", json={"features": [5.1, 3.5, 1.4, 0.2]})

pytest tests/ -v
locust -f locustfile.py --host=http://localhost:8000 --users 100 --spawn-rate 10

The Standard You're Aiming For

A production-grade ML codebase should let a new engineer clone the repo, run one setup command, and have tests, linting, and a working local pipeline — no tribal knowledge required, no "ask Sarah how the data loads." Every practice above exists to make that true.

Next: Data Engineering & Versioning — MLOps starts with data, and data needs the same engineering discipline as code.