Kubernetes

Docker solves "run this container the same way everywhere." Kubernetes solves the next problem: running hundreds of containers across a fleet of machines, restarting the ones that crash, scaling them under load, and routing traffic to healthy instances — none of which one docker run command can do.

The Cluster Itself: Control Plane and Worker Nodes

Before any object (Pod, Deployment, Service) makes sense, it helps to see the real machine topology those objects get scheduled onto — a Kubernetes cluster is two kinds of machine working together, each running a small, fixed set of components.

What is it? A cluster is split into a control plane (the brain — decides what should run where) and any number of worker nodes (the muscle — actually run the containers).

How does it work? Four control-plane components, each with one job: kube-apiserver is the front door every request goes through (including kubectl itself) — it handles auth and admission control, then reads and writes cluster state in etcd, the cluster's source-of-truth key-value store. kube-scheduler watches for Pods that don't have a node assigned yet and picks one. kube-controller-manager runs the reconciliation loops — the exact mechanism the diagram below (desired vs. actual replica count) visualizes; this is the component actually doing that work. Every worker node runs three components: kubelet, the node's own agent, watches the API server for Pods assigned to it and tells the container runtime (containerd or CRI-O — the same OCI-standardized chain Docker uses, reached through the Container Runtime Interface, not a runtime-specific API) to start or stop them; kube-proxy maintains the network rules on that node that make Service routing actually work.

Why is it useful? This is the real, complete request flow behind every kubectl apply: the API server authenticates the request and writes the desired state to etcd → the scheduler notices an unscheduled Pod and picks a node → that node's kubelet starts the container through the runtime → kube-proxy updates routing so a Service can actually reach it. Every object described below is a declaration written to etcd through the API server — the control plane and worker-node components above are what actually turns that declaration into a running, reachable container.

Fundamentals

• Pod: the smallest deployable unit — one or more tightly-coupled containers sharing network/storage, scheduled together onto a node.
• Deployment: declares the desired state for a set of Pods (which image, how many replicas) — Kubernetes continuously reconciles reality toward that desired state, replacing crashed Pods automatically.
• Service: a stable network endpoint in front of a changing set of Pods — Pods come and go (and get new IPs) as they scale and restart, a Service gives callers one address that doesn't.
• Namespace: a logical partition of a cluster — separating dev/staging/prod, or separating teams, within one physical cluster.
• ConfigMap / Secret: externalized configuration and sensitive values respectively, injected into Pods at runtime rather than baked into the image (the same principle as Engineering Foundations's config management, at the cluster level).
• Volume / PV / PVC: Kubernetes's storage abstraction — a PersistentVolume is actual storage, a PersistentVolumeClaim is a Pod's request for some of it, decoupling "what storage exists" from "what a given Pod needs."
• Ingress: routes external HTTP(S) traffic into Services inside the cluster, based on hostname/path rules — the cluster's front door.
• Job / CronJob: run-to-completion workloads (a batch inference job, a nightly retraining job) rather than long-running services — the Kubernetes-native equivalent of a scheduled pipeline step.

A real Deployment + Service pair for a model-serving container built in Containers — three replicas, explicit resource requests/limits, behind a stable ClusterIP:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: model-api
  template:
    metadata:
      labels:
        app: model-api
    spec:
      containers:
        - name: model-api
          image: my-registry/my-model-api:1.0
          ports:
            - containerPort: 8000
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "1"
              memory: "2Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: model-api
spec:
  selector:
    app: model-api
  ports:
    - port: 80
      targetPort: 8000

kubectl apply -f deployment.yaml
kubectl get pods -l app=model-api
kubectl describe pod model-api-<hash>          # scheduling/events for one Pod
kubectl logs -f deployment/model-api           # follow logs across all replicas
kubectl exec -it deployment/model-api -- bash  # shell into a running Pod

kubectl rollout status deployment/model-api    # watch a rollout finish
kubectl rollout undo deployment/model-api      # roll back to the previous revision
kubectl scale deployment/model-api --replicas=5

Advanced

• Helm: a package manager for Kubernetes — templated YAML bundled into reusable "charts," so deploying a complex multi-resource application is one helm install instead of hand-writing dozens of YAML files.
• StatefulSets: like a Deployment, but for Pods that need stable identities and storage (databases, anything that isn't safely interchangeable) — regular Deployments assume Pods are stateless and disposable.
• DaemonSets: ensures exactly one copy of a Pod runs on every node — used for node-level agents (log collectors, monitoring agents, GPU device plugins).

• HPA (Horizontal Pod Autoscaler): automatically scales the number of Pod replicas based on observed metrics (CPU, memory, or custom metrics like queue depth/requests-per-second) — the mechanism behind "scale up under load, scale down when it's quiet."

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: model-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: model-api
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

kubectl apply -f hpa.yaml
kubectl get hpa model-api --watch    # observe replica count react to load in real time

• Resource requests/limits: declare how much CPU/memory a Pod needs (requests, used for scheduling) and the hard ceiling it can't exceed (limits) — get this wrong and either the scheduler wastes capacity or a Pod gets OOM-killed under real load.

• RBAC (Role-Based Access Control): who/what can do what inside the cluster — the Kubernetes-level analog to IAM.
• Network policies: firewall rules for which Pods can talk to which other Pods — default Kubernetes networking is flat and open, network policies restrict it.

• Operators: custom controllers that encode operational knowledge for running a specific piece of software on Kubernetes (e.g. a Postgres Operator that knows how to fail over and back up itself) — the mechanism behind tools like KServe (see APIs & Model Serving).

Why This Matters for ML Specifically

GPU scheduling adds a wrinkle regular web workloads don't have: GPUs are expensive, often can't be fractionally shared cleanly, and Pods need explicit nvidia.com/gpu resource requests plus the NVIDIA device plugin (a DaemonSet) running on every GPU node. Autoscaling a GPU-backed inference Deployment under HPA is also slower and costlier than autoscaling a CPU web service — GPU nodes are expensive to keep idle and slow to provision on demand, which is a major reason serverless-style GPU platforms (see LLM Hosting & Serving Patterns) exist as an alternative to running your own GPU-backed Kubernetes cluster.

The resource request that actually reserves a GPU for a Pod (this replaces the resources block in the Deployment above for a GPU-backed inference workload):

resources:
  requests:
    nvidia.com/gpu: 1
  limits:
    nvidia.com/gpu: 1   # GPUs can't be overcommitted like CPU/memory -- request == limit

kubectl get nodes -o json | jq '.items[].status.allocatable."nvidia.com/gpu"'  # GPU capacity per node
kubectl describe node <gpu-node-name>   # confirm the NVIDIA device plugin advertised GPUs as allocatable

Next: Infrastructure as Code — everything above should be defined in version-controlled code, not clicked together by hand.