CI/CD & ML CI/CD

Continuous Integration and Continuous Deployment automate the path from "code changed" to "that change is safely running in production." ML systems need everything regular CI/CD does, plus checks that don't exist in ordinary software at all — because a model can be syntactically fine and still be wrong.

CI vs. CD

• CI (Continuous Integration): on every push/PR, automatically test, build, and lint the code, and scan it for known vulnerabilities — catching breakage before it merges, not after.
• CD (Continuous Deployment/Delivery): on a successful CI run (often only from the main branch), automatically deploy the change, run post-deploy checks, and roll it out to production — removing manual, error-prone release steps.

GitHub Actions is the most common CI engine — YAML workflows triggered by git events. Argo CD is a common CD tool specifically for Kubernetes: it continuously reconciles a cluster's actual state to match what's declared in a git repo ("GitOps") — deploying by merging a PR rather than running kubectl apply by hand.

A real GitHub Actions CI workflow — lint, test, and build a container image on every push:

name: CI
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: ruff check .
      - run: pytest tests/ --cov=src

  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: my-registry/my-model-api:${{ github.sha }}

Argo CD's side of GitOps is a kubectl apply on its own Application resource, not a deploy script — Argo CD then continuously reconciles the cluster toward whatever's in the target repo/path from that point on:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: model-api
spec:
  source:
    repoURL: https://github.com/my-org/model-api-manifests
    path: overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Why ML CI/CD Needs More Than Code Checks

Regular CI/CD assumes: if the code passes tests and builds, it's safe to ship. That assumption breaks for ML, because the artifact that actually matters — the trained model — is a function of code and data, and "passes unit tests" says nothing about whether the model is still accurate on real inputs. ML CI/CD adds:

• Drift checks: does the incoming data still look like the data the model was trained/validated on? (See Monitoring & Drift Detection for the actual math.) A code change can pass every test and still ship a model quietly degrading against data that's moved.
• Model regression testing: run the candidate model against a fixed, held-out benchmark dataset and assert its metrics haven't dropped below a threshold — the ML equivalent of a regression test suite, because a model "passing" isn't binary the way a function returning the right value is.

• Automated retraining triggers: a pipeline that retrains automatically on a schedule, or when drift crosses a threshold, and only promotes the new model if it beats the current production model on the benchmark — closing the loop from Pipeline Orchestration and Deployment Strategies into something that runs itself.

What a Real ML CI/CD Pipeline Checks, End to End

1. Code passes lint/unit/integration tests (standard CI).
2. New/changed data passes validation (schema, ranges — see Data Engineering & Versioning).
3. A training run completes and logs to the experiment tracker (see Experiment Tracking).
4. The resulting model beats the current production model's metrics on a fixed benchmark — otherwise the pipeline stops here, automatically.
5. The model is registered (see Feature Stores & Model Registry) and containerized (see Containers).
6. The new version is deployed using a safe rollout strategy (see Deployment Strategies), not an instant full-traffic swap.
7. Post-deploy monitoring confirms the new version is healthy before the rollout completes.

Every step above exists because skipping it has a specific, real failure mode — a pipeline that only does step 1 is doing software CI/CD wearing an ML costume. Steps 3-4 (train, then gate on the benchmark) as an actual job, not just a description:

  train-and-gate:
    needs: test
    runs-on: [self-hosted, gpu]
    steps:
      - uses: actions/checkout@v4
      - run: python train.py --config configs/prod.yaml   # logs to the experiment tracker as it runs
      - run: python evaluate.py --run-id ${{ steps.train.outputs.run_id }} --benchmark data/benchmark.parquet
      # evaluate.py exits non-zero if the candidate's metrics don't beat production's --
      # a failing step here stops the workflow before anything downstream (register, deploy) runs

Next: Feature Stores & Model Registry — the systems that make steps 2 and 5 above actually queryable and reusable, not just "it happened once."